Shifna Zarnaz
4 min read

Storing and Retrieving an Array of Keys in S3Bucket with Go and AWS SDK

Introduction:

In this blog post, we will explore how to store and retrieve an array of keys in Amazon S3 (Simple Storage Service) using the Go programming language and the AWS SDK. We’ll analyze a code snippet that demonstrates the process of storing and retrieving an array of keys from an S3 bucket and discuss its implementation details.

Storing an Array of Keys in S3:

The code snippet provides a method named StoreArrayKeysInS3 that allows you to store an array of keys in S3. Let's break down the implementation:

Generate array of values to be stored in S3 : The method dummymethod is called to retrieve keys of type array of strings and a root token of type string. These keys will be stored in S3.

                  
                    keys, roottoken, _ := v.dummymethod()

2. Set up AWS SDK Go session: An AWS session is created using the provided regionregion, accessKeyID, and secretAccessKey. These credentials are used to authenticate with AWS services. 

                  
                    awsSession, err := session.NewSession(&aws.Config{
                      Region:      aws.String(region),
                      Credentials: credentials.NewStaticCredentials(accessKeyID, secretAccessKey, ""),
                     })
                     if err != nil {
                      return nil, err
                     }

3.Create an S3 service client: A new S3 service client is instantiated using the AWS session. This client will be used to interact with the S3 API.

                  
                    s3Svc := s3.New(awsSession)

4.Prepare the keys for storage: The keys generated earlier are appended to the keys slice. Then, the keys are joined into a single string using newlines as separators. This combined string will be stored in S3.

                  
                    keys = append(keys, roottoken)
                    // Join the strings into a single string with newlines
                    combinedString := bytes.Join([][]byte{[]byte(strings.Join(keys, "\n"))}, []byte("\n"))

5.Upload the string to S3: The PutObject method of the S3 service client is called to upload the combined string as an object to the specified S3 bucket and object key. The string data is wrapped in a bytes.Reader before being uploaded.

                  
// Upload the string to S3
 _, err = s3Svc.PutObject(&s3.PutObjectInput{
  Bucket: aws.String(bucketName),
  Key:    aws.String(objectKey),
  Body:   bytes.NewReader(combinedString),
 })
 if err != nil {
  return nil, err
 }

6. Return the keys: If the upload is successful, the function returns the keys slice. This allows the caller to access the stored keys. If an error occurs, it is returned.

                  
                    func (v vault) StoreStringsInS3(objectKey, bucketName, region, accessKeyID, secretAccessKey string) ([]string, error) {
                      // Set up the AWS SDK Go session
                      keys, roottoken, _ := v.dummymethod()
                      awsSession, err := session.NewSession(&aws.Config{
                       Region:      aws.String(region),
                       Credentials: credentials.NewStaticCredentials(accessKeyID, secretAccessKey, ""),
                      })
                      if err != nil {
                       return nil, err
                      }
                     
                      // Create an S3 service client
                      s3Svc := s3.New(awsSession)
                      keys = append(keys, roottoken)
                      // Join the strings into a single string with newlines
                      combinedString := bytes.Join([][]byte{[]byte(strings.Join(keys, "\n"))}, []byte("\n"))
                     
                      // Upload the string to S3
                      _, err = s3Svc.PutObject(&s3.PutObjectInput{
                       Bucket: aws.String(bucketName),
                       Key:    aws.String(objectKey),
                       Body:   bytes.NewReader(combinedString),
                      })
                      if err != nil {
                       return nil, err
                      }
                     
                      return keys, nil
                     }

Retrieving an Array of Keys from S3:

The code snippet also provides a method named RetrieveArrayKeysFromS3 that retrieves an array of keys from an S3 bucket. Let's examine its implementation:

1. Set up AWS SDK Go session: An AWS session is created using the provided credentials.

                  
                    awsSession, err := session.NewSession(&aws.Config{
                      Region:      aws.String(region),
                      Credentials: credentials.NewStaticCredentials(accessKeyID, secretAccessKey, ""),
                     })
                     if err != nil {
                      return nil, err
                     }

2. Create an S3 service client: A new S3 service client is instantiated using the AWS session.

                  
                    s3Svc := s3.New(awsSession)

3. Retrieve the object from S3: The GetObject method is called with the specified bucketName and objectKey to retrieve the object containing the stored keys from S3.

                  
                    resp, err := s3Svc.GetObject(&s3.GetObjectInput{
                      Bucket: aws.String(bucketName),
                      Key:    aws.String(objectKey),
                     })
                     if err != nil {
                      return nil, err
                     }

4. Read the string data: The response body is read using ioutil.ReadAll to obtain the string data. The resp.Body is closed using defer to ensure proper cleanup. 

                  
                    // Read the string data from the response body
                    stringData, err := ioutil.ReadAll(resp.Body)
                    if err != nil {
                     return nil, err
                    }

5. Split the string into an array: The obtained string data is split into an array of strings using newlines as separators. If the last element is an empty string, it is removed from the array.

                  
                    // Split the string into an array of strings using newlines as separators
                    strings := strings.Split(string(stringData), "\n")
                    if len(strings) > 0 && strings[len(strings)-1] == "" {
                     strings = strings[:len(strings)-1]
                    }

6. Return the keys: The function returns the array of keys obtained from the S3 object. If successful, the returned array will exclude any trailing empty string. If an error occurs during any step of the process, it will be returned.

                  
                    func (v *vault) RetrieveStringsFromS3(bucketName, objectKey, region, accessKeyID, secretAccessKey string) ([]string, error) {
                      // Set up the AWS SDK Go session
                     
                      awsSession, err := session.NewSession(&aws.Config{
                       Region:      aws.String(region),
                       Credentials: credentials.NewStaticCredentials(accessKeyID, secretAccessKey, ""),
                      })
                      if err != nil {
                       return nil, err
                      }
                     
                      // Create an S3 service client
                      s3Svc := s3.New(awsSession)
                     
                      // Retrieve the object from S3
                      resp, err := s3Svc.GetObject(&s3.GetObjectInput{
                       Bucket: aws.String(bucketName),
                       Key:    aws.String(objectKey),
                      })
                      if err != nil {
                       return nil, err
                      }
                     
                      defer resp.Body.Close()
                     
                      // Read the string data from the response body
                      stringData, err := ioutil.ReadAll(resp.Body)
                      if err != nil {
                       return nil, err
                      }
                     
                      // Split the string into an array of strings using newlines as separators
                      strings := strings.Split(string(stringData), "\n")
                      if len(strings) > 0 && strings[len(strings)-1] == "" {
                       strings = strings[:len(strings)-1]
                      }
                     
                      return strings[:len(strings)-1], nil
                     }

Conclusion:

In this blog post, we explored a code snippet that demonstrates how to store and retrieve an array of keys in an S3 bucket using Go and the AWS SDK. We discussed the implementation details of the StoreArrayKeysInS3 and RetrieveArrayKeysFromS3 methods, which utilize the AWS SDK to interact with S3. By understanding this code, you can integrate similar functionality into your Go applications to store and retrieve an array of key data from S3 with ease.

Note: Make sure to handle any errors appropriately in your actual implementation to ensure a robust and reliable system.