Fully Managed D-Wave QBoost on AWS SageMaker
Hi everyone
Hoping to help the community to grow and create production ready applications that solves real business use cases, I wanted to create a tutorial.
Here I try to explain how to train your model in the D-Wave machine, and serve it in a classical API endpoint, everything through AWS SageMaker.
The methodology described in the repository is general and it's meant to be modified, and extended in order to allow many different models and approaches for different problems.
The same procedure can be replicate in Google Cloud Platform using their managed services.
Furthermore I'm planning to create a more complete Medium article where I explain the above example from scratch, and extend it with serverless services like AWS Lambda & API Gateway to have Authentication and Load Balancing managed by the cloud servicer.
(Hint: ... Imaging cloud-agnostic architecture like Kubernetes cluster that use seamlessly CPU / GPU / QPU according the tasks, that automatically scale & optimise load balance. It's coming...slowly :) )
Thank you D-Wave & all the users! Keep rocking the Multiverse!!
Cal
Comments
I like. Auditing and fallback/failover would be nifty additions.
Hi Thomas, thank you for the feedback!
What would you think of:
Fallback/Failover: Automatically go for Scikit DecisionTree or RandomForest --> Is it correct?
Auditing: Can you please elaborate more on this? I'm very curious about it.
Hi Cal,
For everything I say, keep in mind I am coming from the perspective of near-real-time operations in a production environment, almost always on remote unsupervised systems.
For fallback/failover, a compute job always needs to be completed. If it is not completed (in a timely manner), then the system needs to know so it can take corrective action immediately. It does not matter how that compute job is completed. A system might have jobs performed locally on the client, or sent back to a server, or periodically performed on a server (for caching), or sent out to a secondary server if the first server does not have sufficient resources to complete the job in a timely manner, or at all. The implementation details for the compute component do not matter as much as having a reliable interface. For the case of machine learning, choosing between a random forest implementation or a decision tree does not matter. (For ML, the devil is in the details, so the nature of the data is what really drives that decision.)
For auditing, there are two important factors. First, what is the cost accounting for the work that was (or was not) done? That information needs to propagate downstream in the accounting chain. Second, I'm in the field of cybersecurity, which has some shades of law enforcement. Factors such as provable correctness or calculations of certainty are important when considering what actions can be reasonably taken in response to a possible cybersecurity (or criminal) event.
So, I have an interest in what you are working on because of the potential for modularity and the serendipity that comes with mixing technologies.
Hi Thomas,
Thank you for your detailed explanation. I will devise a fallback procedure on a classical Decision Tree, in case of any event that might break the computational chain within Sagemaker.
I also see your point about the response reliability of the overall process, which by nature depends on multiple factors, and this lead to a more complicated solution. For this reason I would try to use the AWS Lambda service, which is a fully managed server-less way to call anything that is deployed on Sagemaker and will take care about the balancing of the request's load. Also on top of this, I would like to put an API Gateway which will handle Authentication and I/O stream. However a stress test benchmark should be performed to assess the quality of the overall architecture.
Regarding the auditing, I really appreciate your comment. The future of ML models, whether quantum or classical, depends on being able to be provable and fully reproducible, because only in this way they can be introduced in critical business cases that are strongly involved in regulatory processes.
Thank you again for your interest, I will post here any further improvements to this project.
Cal
Please sign in to leave a comment.