Skip to main content

4 posts tagged with "web development"

View All Tags

How to Upload a File to S3: A Step-by-Step Guide

· 4 min read

Introduction

Amazon S3 (Simple Storage Service) is a scalable object storage service used widely for various data storage needs. This guide will walk you through the process of uploading a file to an S3 bucket, from creating the bucket and setting up the necessary permissions, to writing and testing the code that performs the upload. Let's get started!

Create an S3 Bucket

First we need to create a bucket in S3.

create a bucket

Create a Policy

Next, we need to create a policy that grants access to this bucket.

  1. Navigate to the IAM section of AWS.

iam dashboard

  1. Go to the policy section in the left sidebar.

policy dashboard

  1. Create a new policy.

create policy

  1. Set the resource to S3 and check "All S3 actions" in the "actions allowed" section.

set resource and actions

  1. In the Resources section, input the ARN (Amazon Resource Name) of your S3 bucket opposite the bucket label. You can find the ARN in the properties section of the bucket.

arn

resource section

add bucket arn

added bucket arn

  1. Scroll down and select "Any" for the object label, which grants permission on any object inside the bucket.

set object label to any

  1. Provide a name and description for your policy.

add details in policy

Create an IAM User

After creating the policy, we need to create an IAM user and assign this policy to the user.

  1. Go to the user section in the IAM Dashboard.

create iam user

  1. Enter a name for the IAM user.

enter name for iam user

  1. Skip the AWS Management console access since we only need programmatic access for this user. Attach the policy created earlier to this user.

attach policy

attach policy

  1. Review the details and create the user.

review iam user

  1. Create access keys for the user so that our API can access the bucket on behalf of the created user. Go to the "Security Credentials" section.

move to security credentials section

  1. Scroll down to "Access Keys" and create a new access key.

scroll down to access key

create access key

  1. Optionally, add a description to the access key. Make sure to save the secret access key as it will not be retrievable later.

add description to access key

access key creds

Create a web server

To handle file uploads, we will create a web server using the Go programming language and the go-fiber package.

  1. Download the required packages:
go get github.com/gofiber/fiber/v2
go get github.com/aws/aws-sdk-go/aws github.com/aws/aws-sdk-go/aws/credentials github.com/aws/aws-sdk-go/aws/session github.com/aws/aws-sdk-go/service/s3/s3manager

download packages

  1. Create a web server listening on port 3000:

create web server

  1. Initialize the AWS SDK:

import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
)

var Uploader *s3manager.Uploader

func NewAWS() {

var region string = "" // AWS Region
var accessKey string = "" // access key
var secretKey string = "" // secret access key

awsSession, err := session.NewSessionWithOptions(
session.Options{
Config: aws.Config{
Region: aws.String(region),
Credentials: credentials.NewStaticCredentials(
accessKey,
secretKey,
"",
),
},
})

if err != nil {
panic(err)
}

Uploader = s3manager.NewUploader(awsSession)
}

Initialize this function in main.go:

package main

import (
"github.com/Xebec19/psychic-enigma/internal"
"github.com/gofiber/fiber/v2"
)

func main() {

internal.NewAWS() // initialize AWS SDK

app := fiber.New()

app.Get("/", func(c *fiber.Ctx) error {
return c.SendString("Hello, World!")
})

app.Listen(":3000")
}

  1. Create a function to upload a file to S3:

type Result struct {
Value string
Err error
}

func UploadImage(file *multipart.FileHeader) <-chan Result {
ch := make(chan Result)

go func() {
defer close(ch)
src, err := file.Open()
if err != nil {
return
}
defer src.Close()

var bucketName string = "" // bucket name

_, err = Uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String(bucketName),
Key: aws.String(file.Filename),
Body: src, // add file body here
})
if err != nil {
ch <- Result{Value: "", Err: err}
return
}

url := fmt.Sprintf("https://%s.s3.amazonaws.com/%s", bucketName, file.Filename)

ch <- Result{Value: url, Err: nil}
}()

return ch
}

  1. Create an API endpoint to handle file uploads:
app.Post("/upload", func(c *fiber.Ctx) error {

form, err := c.MultipartForm()

if err != nil {
return err
}

ch := internal.UploadImage(form.File["image"][0])

response := <-ch
if response.Err != nil {
return response.Err
}

c.SendString(response.Value)
return nil
})

Testing the Code

  1. Use Postman to test the file upload endpoint:

postman request

  1. Verify the file has been uploaded to the S3 bucket:

s3 files

Congratulations! You have successfully uploaded a file to S3. You can find the source code https://github.com/Xebec19/psychic-enigma.

Conclusion

Uploading files to S3 involves creating a bucket, setting up the appropriate IAM policies and user, and writing code to handle the file upload. By following this guide, you should now be able to upload files to S3 using Go and the AWS SDK. Happy coding!

Web development basics

· 4 min read

Overview

Web development is an exciting field that empowers us to create and maintain websites and web applications. It involves leveraging various technologies such as HTML, CSS, and JavaScript to construct the visual and interactive elements of a website.

When it comes to the front-end, we're focused on refining the user interface, ensuring both an aesthetically pleasing design and user-friendly experience. This is where HTML (Hypertext Markup Language) takes the stage, defining the structure of our web content. Meanwhile, CSS (Cascading Style Sheets) steps in to stylize and arrange our content, providing the desired appearance and feel.

On the flip side, the back-end encompasses server-side logic that grants dynamism and functionality to our website. This is where programming languages like Python, Ruby, PHP, or JavaScript (Node.js) come into play. It's also where we manage databases, handling user data and other essential information our website requires.

Now, let's delve into the fundamentals of web browsing and how the internet works.

Browsing the web

As we enter "www.google.com" in our browser's address bar, the query journeys to the ISP (Internet Service Provider), which looks up the query in the DNS (Domain Name Service). ISPs are substantial entities providing internet services to users like us, and DNS acts as a registry mapping domain names to unique IP addresses. These IP addresses serve as distinctive identifiers for each computer connected to the internet.

Upon locating the requested resource in the DNS, the ISP returns the IP address of the server hosting that resource. Subsequently, the server transmits HTML, CSS, and JavaScript files back to us. Once received, these files are executed in our browser, culminating in the display of a complete web page.

searching the web

For further insight into ISPs, you can refer to this informative article on Internet Service Provider Hierarchy.

When we have the IP address, we're able to establish a direct connection with the server. Wondering how we achieve this connection? Refer to the illustrative diagram below:

internet-backbone drawio

Our connection to the ISP initiates a connection to the Internet Backbone, which in turn links us to the server. The Internet Backbone is essentially the core of the internet, comprising high-speed networks interlinked by fiber-optic connections and efficient routers. You can explore the intricate network of the Internet Backbone through this submarine cable map

Introduction to html, css and javascript

Let's now explore a basic example of HTML, CSS, and JavaScript integration.

To create an HTML file, we begin by saving a file with a ".html" extension. This file should open with a:

<!DOCTYPE html>

tag, indicating adherence to the HTML5 standard. Within the file, the root element is the <html> tag, housing two crucial tags: <head> and <body>. The <head> section contains metadata about the webpage (e.g., title, keywords, description), while the <body> section houses the webpage's content.

Here's a simplified example using an <h1> tag to display a heading:

<!doctype html>
<html>
<head>
<title>Web page</title>
</head>
<body>
<h1>Hello world</h1>
</body>
</html>

Screenshot from 2023-06-18 20-37-14

However, a plain webpage lacks visual appeal. This is where CSS steps in, offering the means to introduce colors and other stylistic elements that enhance interactivity. Don't worry about the syntax just yet; we'll delve into that in upcoming articles. Here's a simple demonstration of how we'd create basic styling:

.root {
background-color: blue;
}

This styling can then be applied to our document:

<!doctype html>
<html>
<head>
<title>Web page</title>
<style>
.root {
background-color: blue;
}
</style>
</head>
<body>
<h1 class="root">Hello world</h1>
</body>
</html>

As you'll notice, we've assigned a class property to the <h1> tag. We'll explore this concept further in our CSS discussions.

For interactivity, let's add a button that displays a message box when clicked:

<button onclick="handleClick()">Click me</button>

<script>
function handleClick() {
prompt("hello world");
}
</script>

Incorporating the above code into our webpage:

<!doctype html>
<html>
<head>
<title>Web page</title>
<style>
.root {
background-color: blue;
}
</style>
</head>
<body>
<h1 class="root">Hello world</h1>
<button onclick="handleClick()">Click me</button>

<script>
function handleClick() {
prompt("hello world");
}
</script>
</body>
</html>

Screenshot from 2023-06-18 20-50-44

Conclusion

In the vast landscape of web development, we've uncovered the core mechanics of crafting captivating websites and applications. Armed with HTML, CSS, and JavaScript, we've navigated the blend of structure, style, and interactivity that defines modern web experiences.

As you embark on your web development journey, remember that every line of code is a step toward mastery. Embrace the evolving nature of the field, stay curious, and let your passion drive your progress. The world of web development holds endless opportunities for innovation and growth.

May your coding be crisp, your designs engaging, and your journey fulfilling. Happy coding.

Database Migration with Soda

· 6 min read

Introduction

In the world of software development, effectively managing database schema changes is crucial as applications evolve over time. To ensure a smooth transition between different versions of the database schema, it's essential to use database migration tools that provide a systematic way to apply and track these changes. In this article, we'll explore how to set up database migration for a Go application using the popular migration tool called "Soda." Whether you're an experienced Go developer or new to the language, we'll guide you through the process in a clear and concise manner. Let's dive in!

Photo by Jan Antonin Kolar on Unsplash

Prerequisites

Before we begin, having some knowledge of the Go programming language would be helpful. Additionally, make sure you have Go installed on your system and a working database (e.g., PostgreSQL, MySQL) to which you want to apply migrations.

Steps

Step 1: Project Initialization

To get started with database migrations, let's first initialize our Go project. Open your terminal or command prompt and navigate to the directory where you want to create your project. Run the following command:

go mod init example.com/myproject

For example:

go mod init github.com/Xebec19/soda-example

This command initializes a new Go module for our project.

Step 2: Installing Soda

Soda is a powerful database migration and query library for Go. To install Soda, execute the following command in your terminal:

 go install github.com/gobuffalo/pop/v6/soda@latest

You can find more information about Soda link

Step 3: Setting up Database Connection

Before we proceed, let's ensure we have a database ready. In this example, we'll assume a database named "sodademo" already exists.

screenshot

Next, we need to create a database.yml file to specify the connection details:

development:
dialect: postgres
url: "postgresql://username:your_password@127.0.0.1:5432/sodademo"
pool: 5

production:
url: "postgres://username:your_password@127.0.0.1:5432/sodademo"

In the above configuration, make sure to replace <your_password> with the actual password for your database user. Also, feel free to adjust the address and other settings based on your environment (e.g., development, production).

Step 4: Creating a Migration

Now, let's create our first migration. Run the following command:

soda generate sql migration-name

Replace <migration_name> with a descriptive name for your migration. This command will generate two migration files: an "up" migration file and a corresponding "down" migration file. The "up" file contains the changes we want to apply to the database schema, while the "down" file specifies how to revert those changes.

For the "up" migration, let's create a table:

create table users (

user_id serial primary key,

first_name text not null,

last_name text,

email varchar not null unique,

phone integer,

password varchar not null,

created_on timestamptz DEFAULT now(),

updated_on timestamptz DEFAULT now(),

status varchar(10) default 'active'

)

For the "down" migration, let's drop the table:

drop table users;

Step 5: Applying Migrations

Now it's time to apply the migrations and update our database schema. Execute the following command:

soda migrate

This command will execute all pending migrations. You can find more information about migration commands here.

Let's run our migration file.

screenshot

After running the "up" migration script, the table will be created in our database.

screenshot

Step 6: Rolling Back Migrations

If we need to roll back the last applied migration, we can use the following command:

soda migrate down

Executing the "down" script allows us to revert the changes made by the "up" script.

screenshot

Now, if we check our database, the user table has been deleted.

screenshot

Conclusion and Troubleshooting

Congratulations on successfully setting up database migration for your Go application using Soda! As you continue working with migrations, it's important to be aware of potential issues and how to resolve them. Here are a few common scenarios and frequently asked questions to help you troubleshoot:

1. Migration Errors

If you encounter errors while running migrations, double-check the following:

  • Ensure that your database connection details in the database.yml file are accurate.
  • Verify that you have the necessary permissions to perform database operations.
  • Review the migration files for any syntax errors or conflicting changes.

2. Rollback Issues

In case you face difficulties when rolling back migrations, consider the following:

  • Ensure that the "down" migration file is correctly written to revert the changes made by the corresponding "up" migration.
  • Check if there are any dependencies between migrations that need to be handled properly to ensure successful rollback.

3. Data Integrity

During migrations, it's important to maintain data integrity. Here are a few tips:

  • Handle data transformations carefully to prevent data loss or inconsistencies.
  • Make use of transaction blocks to ensure atomicity and rollback changes if necessary.
  • Backup your database regularly to avoid irreversible data loss in case of unforeseen issues.

4. Complex Database Operations:

Some complex database operations, such as modifying certain constraints, data type conversions, executing raw SQL scripts, or installing plugins like uuid-ossp, may not be suitable for migration files. In such cases, it's recommended to perform these operations manually or explore alternative approaches.

Note: If you would like to explore an example project that demonstrates the concepts discussed in this article, you can find it here.

Frequently Asked Questions (FAQs)

Here are some common questions and their answers to help address any concerns you may have:

Q: Can I add additional fields to an existing table using migrations?

A: Yes, you can create a new migration and use the appropriate syntax to add columns to an existing table. Make sure to handle the necessary data transformations if required.

Q: What if I need to modify or remove a migration after it has been applied?

A: It's generally recommended not to modify or remove migrations that have already been applied to a production database. Instead, create a new migration that addresses the changes you need.

Q: How can I handle database migrations in a team setting?

A: To manage migrations in a collaborative environment, ensure that all team members are working with the same set of migration files and that changes are communicated effectively. Consider using version control systems like Git to track and merge migration changes.

Q: Can I automate the execution of migrations in a deployment pipeline?

A: Absolutely! You can integrate the migration execution as part of your deployment process. Create scripts or use tools like CI/CD pipelines to ensure migrations are applied seamlessly in different environments.

Remember, it's always beneficial to refer to the official documentation and seek support from the community if you encounter any difficulties during your migration journey.

Happy migrating!

HTML Server Sent Events

· 2 min read

Why HTML SSE ?

It always make me curious that how codeforces provide real time notifications to users during contests. Does it use any tool like firebase to set pub/sub ? Does it use long polling ? Sending repeat requests to server to fetch notifications does not sound like a bright idea. Though I am still not very clear about how codeforces handles all those timely prompts but I have came across a way which might serve the purpose and that is HTML Server Sent Events.

What is HTML SSE ?

As mentioned by w3schools "Server-Sent Events (SSE) allow a web page to get updates from a server". Allright! we do not need to send repeated request to the server for fetching update. We can use Server-Sent Events through which our server will send update to the website. Before we dig further, let's have a look on the list of browsers which supports Server-Sent Events.

Implementation

Let's move to the fun part and learn how to implement SSE. First we will create a nodeJs application and create a route as follows:

index.js

As we can see we only need to set "Content-Type" header as "text/event-stream" and send our data. In our case we will be sending a timestamp. Now lets create a simple html page which will receive updates from our server.

index.html

In the above code, we have created an EventSource which is connected to "http://localhost:3000/sse" route. We have used onmessage event to get messages. Some other events are as follows:

  1. onopen: When a connection to the server is opened
  2. onmessage: When a message is received
  3. onerror: When an error occurs

Let's start our server with below command.

node index.js

Now, we will checkout our web page.

page.html

Awesome! We are automatically receiving those timestamps from our server. Hope you find this article helpful. You can find the source code on github. Happy coding!