Just how to delete individual information in a AWS information lake
One article of GDPR is the “right to erasure” or “right to be forgotten” which might require you to implement a solution to delete specific users’ individual information.
Every architecture, regardless of the problem it targets, uses Amazon Simple Storage Service (Amazon S3) as the core storage service in the context of the AWS big data and analytics ecosystem. Despite its versatility and show completeness, Amazon S3 doesn’t come with an way that is out-of-the-box map a user identifier to S3 keys of items which contain user’s data.
This post walks you by way of a framework that can help you purge individual individual data in your organization’s AWS hosted data pond, as well as an analytics solution that uses different AWS storage levels, along side sample rule targeting Amazon S3.
Reference architecture
To address the task of applying a data purge framework, we paid off the issue towards the simple use situation of deleting a user’s information from the platform that makes use of AWS for the information pipeline. The after diagram illustrates this usage situation.
We’re introducing the basic notion of building and keeping an index metastore that monitors the place of each and every user’s documents and allows us find for them effectively, reducing the search r m.
You should use the following architecture diagram to delete a specific user’s data in your organization’s AWS data lake.
Each task to a fitting AWS service for this initial version, we created three user flows that map
Flow 1 Real-time metastore upgrade
The S3 ObjectCreated or ObjectDelete events trigger an AWS Lambda function that parses the object and executes an operation that is add/update/delete keep consitently the metadata index up to date. Read more
