This article is about how I upgraded my previous cron scheduler service that I built in Go for scaling it to around 1 million daily scheduled jobs. This lays out the overall system design, approach and considerations I took in building this system.

The Problem

I had created a cron scheduler service which uses robfig/cron package to schedule cron jobs on the machine. This is good for limited number of jobs, but when the job numbers are increased to thousands and even millions, this system will not work.

As for scheduling the crons, the package uses goroutines and for this huge amount of jobs, it can easily overwhelm the hardware resource. Now there can be a solution to run multiple scheduler instances and divide the overall crons on these instances. But this brings another issue. When crons are scheduled on multiple machines, what if an instance suddenly crashes? The jobs scheduled on that machine will be lost!. We will then have to build some tracking mechanism to track each cron respective to the machine, which will bring further complexities. So have the job’s state on a server instance would not work.

So in this article, we discuss how to make a system that can have million jobs scheduled daily!

The Solution

I have referred multiple system design blogs & videos(Mayil’s Medium, Algomaster, Jordan’s YT) to design a distributed scheduler service, and came up with following points:

• A job submission service(UI/API) will create new jobs and store them in a job store.

• The job store will persist all of the job related data and the run history of the jobs with their status.

• A scheduling service periodically queries the jobs table to get all tasks to run at current time.

• These jobs will be then pushed to a distributed queue.

• These jobs then will be consumed & processed by workers, which will update their job run status in job run history store.

The idea is that, instead of my approach of scheduling jobs on a machine, we can just poll the jobs table every minute to check all jobs that are required to run at that minute.

This is the big change! Instead of actually scheduling hardware cron jobs, we can just fetch from the job store every minute and process them.

After polling and fetching required jobs from DB, we can just publish them to the queue.

For this, the blogs have suggested using the NoSQL Cassandra DB due to it’s low latency and high read-write throughputs that fits well when we have to periodically poll on the DB.

For queueing jobs, we can use Kafka.

My Approach & Considerations

To keep our infra simple and considering limited resources, we will use:

• PostgreSQL DB for storing jobs data and job run history.

• Redis for queuing jobs in redis streams, managing distributed locks and storing job in sorted sets.

Since we are using PostgreSQL DB, we should consider the limitations of polling the DB every minute. Polling periodically can put heavy load on the DB and can increase the I/O ops thus affecting our single point of source for other operations.

To reduce load on DB, we can process jobs in batches. In this implementation, we will query the DB every 10 minutes to load all of the jobs for next 10 minutes temporarily. After 10 minutes, they will be removed from this temporary storage.

Now next question is where should we store the next 10 minute jobs efficiently?

Answer is Redis’s sorted sets!

Redis ZSET

First i had considered Redis lists, but going through the list for specific timestamp can take O(N) time. Which can be huge, considering a large number of jobs for a minute(e.g thousands of jobs per minute at spike).

Using Redis sorted set brings the advantage of scores! In sorted sets, the values are by default ordered based on scores. So we can use the timestamps as scores and this can bring faster access to the jobs when we try to fetch for specific timestamp.

For reading, it will take O(log N + M), where N is the total number of elements in set and M is the total elements returned. So internally, Redis first searches for the first element which satisfies range condition i.e >=min which takes O(log N) time like binary search. And once the node position is found, it takes O(M) time to linearly go through the M items and return them until the range is <= max.

Redis Stream & Consumer Group

For queuing jobs, we will use Redis streams. The workers will be a part of consumer group listening over this stream. Using consumer groups solves the problem of duplicate job execution, since the consumer group itself acts as a load balancer for multiple consumers in the group and distributes the job coming from the stream properly ensuring a job will be delivered to only one consumer.

Redis streams also allows to acknowledge the job messages using XACK, using which we can build our acknowledge and retry mechanism.

When a consumer gets a jobs and suddenly crashes without acknowledging, Redis allows us to claim back that job from a pending entries list. So any other consumer can claim, process and ack that job if the job has been there for too long in the pending list.

Architecture

This system consists of following components:

1. Dashboard: The dashboard lets users create a new job, list all jobs and view job run entries with their status (completed, failed, permanently_failed). The dashboard interface services are served with a load balancer. On creation of a new job, an entry is made in the jobs table.

2. Scheduler: The scheduler service has two sub-service - Batch processor and Publisher.
   The Batch Processor polls the PostgreSQL DB every 10 mins with a Redis lock to query jobs for next 10 min range. After getting the jobs, it adds them to Redis’s sorted set (ZSET).
   The Publisher runs every minute to fetch all jobs from ZSET for current time. It then publishes these jobs to Redis stream.

3. Worker: The worker services are part of a consumer group which gets jobs from the Redis stream and process them further. On success, it makes entry in the job_runs table as completed. On failure, it puts back the jobs to the stream for further retrying. On retrying for max 3 times, it marks the job as permanently failed and creates entry in the job_runs table.

4. PostgreSQL DB: PostgreSQL is used to store the job details in jobs table which is partitioned using the hour time of jobs. e.g a partition for storing jobs whose scheduled hour is 00 to 06, another for 07 to 12, etc. It stores the hour, minute, payload and retry count.
   It has another table job_runs to store the job run entries with their statuses (completed, failed, permanently_failed). It also stores the output (for completed runs) and error (for failed runs).

5. Redis: Redis is used for letting the Batch Processor of scheduler service to create a lock so that only one instance polls the DB.
   Redis’s sorted set is used to store jobs with their timestamp as scores for fast access during every minute polling of the Publisher of scheduler service.
   Redis Stream is used to queue the jobs.

As this is a time sensitive system, storing and processing of jobs are done at their UTC times across all services and DB.

Dashboard

The dashboard lets users create a new job, list all of the jobs and view run entries of a job.

1. Creating & Viewing jobs: When a job is created from dashboard interface, the timezone of the client is also passed with the request. The schedule time is then converted to UTC and the job is inserted into jobs table. The jobs table has id, hour UTC converted, minute UTC converted, type (ping, email, slack, webhook), payload, retries count, created_at and updated_at times.
   The jobs table is partitioned using the hour column in different hour range partitions. e.g jobs_00_to_06 for storing jobs whose UTC hour ranges between 00 and 06, jobs_07_to_12 for storing jobs whose UTC hour ranges between 07 and 12, etc.

2. Viewing job run entries: Job run entries are viewed using the job_runs table which stores the id, job_id, status (completed, failed, permanently_failed), output for success runs, error for failed runs, scheduled_at and completed_at time.

Scheduler service

The Scheduler service has two main components: Batch Processor and Publisher:

Batch Processor:

1. It polls the PostgreSQL DB every 10 mins to read jobs from the jobs table.

2. For multiple scheduler service instances, it first tries to create a Redis lock so that only one scheduler instance polls the DB.

3. If the instance is unable to get the lock, it will exit and wait for next timer period.
   The batch processor fetches jobs of next 10 mins at every 10s rounded time i.e 12:00, 12:10, 12:20,… it fetches jobs of next 10 min range i.e at 12:10, it fetches jobs for 12:11 to 12:20 range, at 12:20, it fetches jobs for 12:21 to 12:30, etc.

4. On getting the Redis lock, jobs are queried. It pushes these jobs to the Redis’s sorted set (ZSET).

5. After pushing the jobs, it also clears all previous jobs if they were not processed in past 10 mins from ZSET. These lost jobs can be logged as lost/failed.

Here, ZSET is used because of its score mechanism. The jobs are stored in a sorted manner using their timestamps as score(sorting key). So, the insertion becomes O(log N) and fetching is O(log N + M).
In case of errors, the locked batch processor can run for maximum 3 times for retrying.

Publisher:

1. It runs for every minute to get jobs from the ZSET for that specific time.

2. All jobs of current time are fetched in batches via all of the scheduler instances until there are no jobs available in ZSET for that current time.

3. The jobs are fetched and deleted at same time using a single lua script which runs on the redis server end to avoid same jobs getting fetched from different instances.

4. Then it publishes all of the jobs to Redis stream using XAdd.

Worker service

The worker services are part of a consumer group listening over this stream. Using consumer groups, we can solve the problem of duplicate job execution, since the consumer group ensures that a single job message is delivered to only one consumer.

1. When a worker is started, it first check for the consumer group, and if it is not present, it creates it.

2. Then we get the job from the stream in a batch of N using XREADGROUP. On every read, we wait for 5 seconds until there are N jobs in the stream. As soon as N jobs becomes available in the stream it fetches them. After 5 seconds, if there are less than N jobs, it still fetches them.

3. These jobs are then processed based on their type which is present in their payload. For ping type, we make a GET call, for email type, we try to send an email using mailtrap.io service, for slack type, we make a POST call to given slack url and for webhook type, we make the POST call from given url and body.

4. If the job execution was success, we make an entry in the job_runs table with completed status and reset the retries count of that job to 0 in jobs table. Then that job message is acknowledged using XAck and deleted from the stream using XDel.

5. If the job execution fails, we first get the retries count of that job from jobs table. If the retries count is less than 3, we make an entry in job_runs table with failed status and increment the retries count of that job in jobs table. Then the job message is acknowledged and deleted from stream. Then a new entry of that job is published to the stream for further retries using XAdd.

6. If on failed job execution, we get the retries count >= 3, we make an entry in the job_runs table with permanently_failed status and reset the retries count to 0 for that job in jobs table. Then the job message is acknowledged and deleted from stream.

The worker can get very slow if there is a single process running on the worker machine. So we can make use of workerpools here. e.g running 10 goroutines in the workerpool to keep processing the jobs coming in redis stream concurrently.

Some Calculations, Testing & Observations

Since we are now targeting around a million scheduled jobs daily, so average tasks per minute would be

(1,000,000)/24×60 ≈ 695 jobs/minute.

i.e 6950 jobs on average in our 10 mins of batch processing time. But this is uniform distribution of tasks and in real system, the tasks would not be evenly distributed across time.

Let’s consider a single busiest 10 min range can have around 75K jobs burst where some mins can have spike of around 15K also.

So with these numbers, I tested the system deployed on ECS + Fargate:

I generated a python script using GPT - generate_insert.py which gives me batch insertion query to create a load in the DB for testing. It takes the hour, start minute, end minute, a list of heavy minutes which will have N jobs and N - the spike count that the heavy minutes will have. It is well customizable.

Configurations:

• 1 dashboard instance

• 2 scheduler instances

• 4 worker instances

• Fragate specs: .25 vCPU and .5 GB Memory.

Observations:

• The batch processing of jobs from DB to ZSET and publishing of jobs from ZSET every minute to the stream was very quick. all happened within milliseconds.

• On worker instances, the consuming of jobs was really quick at the start i.e around 1000 - 1500 jobs per second. But as it continued, due to requeuing of failed jobs(whose execution had 5s timeout) and waiting for 5 seconds until it’s failure and with overall 3 max retries for a failed job, the job execution time on worker machines exponentially grew.

The overall processing of this 10 min batch took around 13 mins where 10 goroutines were working in the workerpool on each worker machine, considering there were sudden spikes of 15k to 20k for some mins.

Without workerpools, it took more than 30 mins. So yes, increasing the hardware resource of a worker machine and adding more goroutines can definitely reduce the overall job execution time. Or else, we can also just horizontally scale the worker machines.

Improvements

Batch Processing failures

Considering a case when a scheduler instance fails after getting the lock, the batch of that 10 min range will never be processed.

We have set the lock expiration time for the batch to 2 mins. So to handle this, on completion of a batch processing, we can put a key in redis to mark the completion of that batch (e.g batch_completed_11_20 - i.e for 12:11 to 12:20) with expiration of 10 mins.

So In the batch processor, we can periodically check for every 2 mins if the batch of that 10 min range was processed or not.

Worker instance crashes

When a job is claimed by a consumer in the consumer group, it is also registered in a Pending Entries List (PEL) to keep track what jobs are not yet acknowledged using Xack. So even if the consumer crashes or is gone from the group, those jobs remain in the PEL.

For handling this, we can run a periodic goroutine to get jobs from PEL whose time has crossed a certain threshold time, then claim them using XCLAIM, process them, ack and remove them from the PEL.

Conclusion

So with this approach, we can definitely process thousands of jobs every minute by making the batch processor pick jobs for every 10 mins and publishing them to the redis stream every minute using ZSET’s fast insertion and retrieval.
It is really amazing to design and build a system that can handle such a huge workload. Still this is not fully reliable and fault tolerant, but additions can be built on this to make it one!

Find this project here:

If you find this article helpful, don't forget to hit the ❤️ button.

Check out my website here and feel free to connect.

Happy Coding! 👨‍💻

This project is about a cron scheduler service which schedules cron jobs that are created using a dashboard interface. It publishes the jobs to the consumer service through RabbitMQ job queue, where it acknowledges successful job execution and retries for upto 3 times on job failures. And thus updates the job status in PostgreSQL database for tracking through the dashboard interface.

Some Theory

1. Cron jobs: These are jobs/tasks which are scheduled to run at a specific time. They can be configured to run only at a given specific time, repeat every day at a specific time, repeat every x hours, etc by using cron expressions.

2. RabbitMQ: It is an open-source message broker that allows two services to communicate asynchronously. It provides a queue through which two separate services can communicate by sending messages. This is one of the best communication methods used by distributed systems. We will be using the AMQP (Advanced Message Queuing Protocol) in this project supported by RabbitMQ.

3. AMQP: The Advanced Message Queuing Protocol is an open standard protocol where messages (data) are published through exchanges to queues. The queues are connected with exchanges using bindings which define the exchanges to which queue the data should be sent. Then other service can consume the messages over the queue

Project Architecture

This project will have two main components:

1. scheduler
   This will host our dashboard and schedule the cron jobs. On cron timings, the jobs will be published to the queue.

2. consumer
   This will consume messages from the queue and process them based on the job type. On success, it will register in PostgreSQL DB. On failure, it will try for up to maximum 3 times. If job fails for 3 retries, it will register as permanently_failed in the DB. This will also ACK (acknowledge) or NACK (un-acknowledge and put back to queue) the jobs depending on the results.

Find this project here:

Note:

The dashboard in this project has pages to display basic stats, listing all of the scheduled jobs and listing job run entries with status (running/completed/failed) which are not explained in this blog.

The main purpose of this blog is to only explain the code for core logic of cron job scheduling, publishing to and consuming from RabbitMQ, processing the job based on type (ping, email, slack, webhook) with acknowledgement & retries and registering everything on the PostgreSQL database.

To understand the project structure used here, please refer to this blog: Effective Project Structure for Backend Projects in Go written by me!

Scheduling cron job & Publishing to RabbitMQ

In scheduler service’s apiService.go file, which is located at /scheduler/internal/services/apiService.go

type ApiService struct {
    db        *sql.DB
    cron      *cron.Cron
    rabbitmq  *amqp091.Connection
    mqChannel *amqp091.Channel
}

func (s *ApiService) CreateNewJob(job *models.Job) error {
    // first putting this job in db
    query := `INSERT INTO jobs(cron_expr, type, payload)
              VALUES ($1, $2, $3) RETURNING id;
    `
    payloadJSON, _ := json.Marshal(job.Payload)
    var jobId int
    if err := s.db.QueryRow(query, job.CronExpr, job.Type, payloadJSON).Scan(&jobId); err != nil {
        return err
    }
    job.Id = jobId

    // scheduling a cron job
    id, err := s.cron.AddFunc(job.CronExpr, func() {
        log.Println("running cron job: publishing to rabbitmq")
        q, err := s.mqChannel.QueueDeclare("cron_events", false, false, false, false, nil)
        if err != nil {
            log.Println("failed creating a queue for rabbitmq: ", err.Error())
            return
        }

        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()

        jsonBody, err := json.Marshal(map[string]any{
            "job":  job,
            "time": time.Now().Format("2006-01-02T15:04:05.000000-07:00"),
        })
        if err != nil {
            log.Println("failed creating payload for rabbitmq: ", err.Error())
            return
        }

        if err := s.mqChannel.PublishWithContext(ctx, "", q.Name, false, false, amqp091.Publishing{
            ContentType: "application/json",
            Body:        jsonBody,
        }); err != nil {
            log.Println("failed publishing to rabbitmq: ", err.Error())
            return
        }
        log.Println("event published to rabbitmq!")

    })
    if err != nil {
        // on cron scheduling error, deleting the job created in db
        delQuery := `DELETE FROM jobs WHERE id = $1`
        s.db.Exec(delQuery, jobId)
        return err
    }

    updateQuery := `UPDATE jobs SET cron_id = $1 WHERE id = $2`
    s.db.Exec(updateQuery, id, jobId)
    job.CronId = int(id)
    return nil
}

This is a service function that we have created for our ApiService and called from /api/create API’s handler.
So, we first insert the job into the jobs table

query := `INSERT INTO jobs(cron_expr, type, payload)
              VALUES ($1, $2, $3) RETURNING id;
    `
    payloadJSON, _ := json.Marshal(job.Payload)
    var jobId int
    if err := s.db.QueryRow(query, job.CronExpr, job.Type, payloadJSON).Scan(&jobId); err != nil {
        return err
    }
    job.Id = jobId

After inserting, we update id of the job variable.

Then we schedule the cron job using robfig/cron package. We have loaded the cron instance from this package into our ApiService.

The cron job is scheduled using AddFunc() function from the package, which takes two parameters, the cron expression and the function to run for this cron.

The cron expression we use is:

x y * * *

Where x is minute and y is hour to repeat every day. This cron expression is inside our job model which we access as job.CronExpr.

// scheduling a cron job
    id, err := s.cron.AddFunc(job.CronExpr, func() {
        log.Println("running cron job: publishing to rabbitmq")
        q, err := s.mqChannel.QueueDeclare("cron_events", false, false, false, false, nil)
        if err != nil {
            log.Println("failed creating a queue for rabbitmq: ", err.Error())
            return
        }

Inside the function that will be invoked at cron time,

We create a queue for the RabbitMQ using rabbitmq/amqp091-go package’s QueueDeclare() function.
We have stored the RabbitMQ’s channel inside our ApiService as mqChannel which is of type *amqp091.Channel.

So we create the queue by passing name of the queue which in this case is cron_events, and then check for any errors in creating of the queue.

This creates the queue(if not already exists) on which the consumer will get to consume the messages. Then we prepare the payload to send in the message over queue.

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

jsonBody, err := json.Marshal(map[string]any{
    "job":  job,
    "time": time.Now().Format("2006-01-02T15:04:05.000000-07:00"),
})
if err != nil {
    log.Println("failed creating payload for rabbitmq: ", err.Error())
    return
}

Here, we first created a context with 5 seconds timeout to be used for publishing the message to queue. It is important in case it takes more than 5 seconds to publish the message.

Then we create a json with job key that will store our job model’s json data and time key which stores the current schedule time of this cron job.

Then we check for any errors in creating the payload body and continue to publish the message

if err := s.mqChannel.PublishWithContext(ctx, "", q.Name, false, false, amqp091.Publishing{
    ContentType: "application/json",
    Body:        jsonBody,
}); err != nil {
    log.Println("failed publishing to rabbitmq: ", err.Error())
    return
}

We publish the message to queue using PublishWithContext() function on mqChannel and pass the timeout context, keep the default ““ exchange and provide queue’s name.

In this, we have also passed our payload using amqp091.Publishing{} where we have mentioned the ContentType and Body.

We then check for any errors in publishing the message to queue.

Then after scheduling the cron job and defining the function to run for cron, we check for any errors in scheduling of cron job

if err != nil {
    // on cron scheduling error, deleting the job created in db
    delQuery := `DELETE FROM jobs WHERE id = $1`
    s.db.Exec(delQuery, jobId)
    return err
}

If there were errors, then we delete the DB entry we made for this job earlier.

updateQuery := `UPDATE jobs SET cron_id = $1 WHERE id = $2`
s.db.Exec(updateQuery, id, jobId)
job.CronId = int(id)
return nil

If there were no errors, we finally update the cron_id of our job in the DB’s entry.

Listening on the RabbitMQ’s Job Queue & Consuming Jobs

In consumer service’s rabbitmqService.go file, which is located at /consumer/internal/services/rabbitmqService.go

We have first created a RMQService struct and a NewRMQService() constructor.

type RMQService struct {
    dbService *DBService
    conn      *amqp091.Connection
    channel   *amqp091.Channel
}

func NewRMQPService(db *sql.DB, conn *amqp091.Connection, channel *amqp091.Channel) *RMQService {
    return &RMQService{
        dbService: NewDBService(db),
        conn:      conn,
        channel:   channel,
    }
}

This struct holds the db connection dbService, the RabbitMQ’s connection conn and the RabbitMQ’s channel channel.

In the main.go's main function, we create a new rabbitmq service’s instance

rabbitmqService := services.NewRMQService(cfg.Db, cfg.RabbitMQ, cfg.MQChannel)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
rabbitmqService.Start(ctx)

After creating the rabbitmq service’s instance, we define a WithCancel context and call its cancel() function on defer of main function to gracefully close the RabbitMQ’s connection when the consumer service shuts down. Then we call the Start() function.

func (s *RMQService) Start(ctx context.Context) {
    q, err := s.channel.QueueDeclare("cron_events", false, false, false, false, nil)
    if err != nil {
        log.Println("error in creating rabbitmq queue: ", err.Error())
        return
    }

    msgs, err := s.channel.Consume(q.Name, "", false, false, false, false, nil)
    if err != nil {
        log.Println("error in creating a consume channel for rabbitmq: ", err.Error())
        return
    }

    go func() {
        for {
            select {
            case <-ctx.Done():
                log.Println("Stopping rabbitmq...")
                return
            case msg, ok := <-msgs:
                if !ok {
                    log.Println("RabbitMQ message channel is closed.")
                    return
                }
                processMessage(&msg, s.dbService)
            }
        }
    }()
    log.Println("RabbitMQ Consumer Running: Waiting for messages...")

    <-ctx.Done()
}

In Start function, we first declare the queue using QueueDeclare() and then call the Consume() function.

msgs, err := s.channel.Consume(q.Name, "", false, false, false, false, nil)
if err != nil {
    log.Println("error in creating a consume channel for rabbitmq: ", err.Error())
    return
}

In this, we pass the queue’s name and mentioned autoAck as false. So that we can manually ACK and NACK the job messages for retries.

go func() {
    for {
        select {
        case <-ctx.Done():
            log.Println("Stopping rabbitmq...")
            return
        case msg, ok := <-msgs:
            if !ok {
                log.Println("RabbitMQ message channel is closed.")
                return
            }
            processMessage(&msg, s.dbService)
        }
    }
}()
log.Println("RabbitMQ Consumer Running: Waiting for messages...")

<-ctx.Done()

Then we run a goroutine in which we run an infinite for loop which basically has a select statement which is waiting on the cancel context’s cannel and RabbitMQ’s consume channel.

When the application shuts down, close() of the context is called and a value is received on ←ctx.Done() which returns from this function to close the RabbitMQ’s connection.

When a message is published on the queue by the scheduler service, that message is received on msgs channel which is fetched by ←msgs case and the message is stored in msg

Then we call the processMessage() function to further parse the job’s payload and process the job.

Processing the Job

func processMessage(msg *amqp091.Delivery, dbService *DBService) {
    log.Println("Received message on RabbitMQ channel: ", string(msg.Body))

    // parsing message body which has keys "job"[Job json] and "time"[Schedule time string]
    var body map[string]any
    if err := json.Unmarshal(msg.Body, &body); err != nil {
        log.Println("error in extracting message payload: ", err.Error())
        msg.Ack(false)
        return
    }

    // parsing job json from the message body json
    jobBody, ok := body["job"].(map[string]any)
    if !ok {
        log.Println("error in extracting job json: ", body["job"])
        msg.Ack(false)
        return
    }
    // parsing scheduled time from the message body json
    sTime, ok := body["time"].(string)
    if !ok {
        log.Println("error in extracting scheduled time: ", body["time"])
        msg.Ack(false)
        return
    }

    // converting the job json to bytes, to convert it to models.Job
    jobBodyByte, err := json.Marshal(jobBody)
    if err != nil {
        log.Println("error in parsing job json: ", err.Error())
        msg.Ack(false)
        return
    }
    var job *models.Job
    if err := json.Unmarshal(jobBodyByte, &job); err != nil {
        log.Println("error in extracting job: ", err.Error())
        msg.Ack(false)
        return
    }

    // get retries of any existing job entry for this job id, scheduled time which was not failed
    var jobEntry *models.JobEntry
    j, err := dbService.getExistingJobEntry(job, sTime)
    if err != nil {
        log.Println("error in getting a job entry: ", err.Error())
        msg.Ack(false)
        return
    }
    if j != nil {
        // if a job entry already exists, update that in jobEntry variable
        jobEntry = j
    } else {
        createdEntry, err := dbService.createNewJobEntry(job, sTime)
        if err != nil {
            log.Println("error creating new entry in db: ", err.Error())
            msg.Ack(false)
            return
        }
        jobEntry = createdEntry
    }

    // checking if the retry count reached max retries
    if jobEntry.Retries >= MaxJobRetries {
        dbService.markJobAsPermanentlyFailed(jobEntry)
        msg.Ack(false)
        return
    }

    switch job.Type {
    case "ping":
        if err := processPingJob(dbService, job, jobEntry, sTime); err != nil {
            handleJobError(dbService, err, msg, jobEntry)
            return
        }
    case "email":
        if err := processEmailJob(dbService, job, jobEntry, sTime); err != nil {
            handleJobError(dbService, err, msg, jobEntry)
            return
        }
    case "slack":
        if err := processSlackJob(dbService, job, jobEntry, sTime); err != nil {
            handleJobError(dbService, err, msg, jobEntry)
            return
        }
    case "webhook":
        if err := processWebhookJob(dbService, job, jobEntry, sTime); err != nil {
            handleJobError(dbService, err, msg, jobEntry)
            return
        }
    default:
        handleJobError(dbService, fmt.Errorf("invalid event type: %s", job.Type), msg, jobEntry)
    }

    msg.Ack(false)
}

In this function, we first get the json body from message’s payload. Then we extract the job’s json from this body stored at key job and scheduled time store at key time.

Then we encode the job’s json to convert it to a Job type variable stored in job *models.Job.

If there are errors at any of the parsing step, we acknowledge the message using Ack(false). We pass multiple as false in Ack to avoid acknowledging any other prior deliveries.

We do this, because we can’t process the job further and have to remove this message from the queue by acknowledging.

// get retries of any existing job entry for this job id, scheduled time which was not failed
var jobEntry *models.JobEntry
j, err := dbService.getExistingJobEntry(job, sTime)
if err != nil {
    log.Println("error in getting a job entry: ", err.Error())
    msg.Ack(false)
    return
}
if j != nil {
    // if a job entry already exists, update that in jobEntry variable
    jobEntry = j
} else {
    createdEntry, err := dbService.createNewJobEntry(job, sTime)
    if err != nil {
        log.Println("error creating new entry in db: ", err.Error())
        msg.Ack(false)
        return
    }
    jobEntry = createdEntry
}

Then we call getExistingJobEntry() function from dbService to check if there were any previous execution trials for this job at that specific scheduled time. If there was prior execution and this is a retry round, then we initiate the jobEntry variable.

If there were no previous execution, we create a new job entry with running status for that scheduled time in database using createNewJobEntry() function and put it’s value in jobEntry variable.

// checking if the retry count reached max retries
if jobEntry.Retries >= MaxJobRetries {
    dbService.markJobAsPermanentlyFailed(jobEntry)
    msg.Ack(false)
    return
}

Then we check in case of retrying job, if the retry count has exceeded MaxJobRetries i.e 3.

If yes, then we update that job with permanently_failed status in database for that scheduled time using markJobAsPermanentlyFailed() function and acknowledge the message to remove it from the queue and finally return from the function.

switch job.Type {
case "ping":
    if err := processPingJob(dbService, job, jobEntry, sTime); err != nil {
        handleJobError(dbService, err, msg, jobEntry)
        return
    }
case "email":
    if err := processEmailJob(dbService, job, jobEntry, sTime); err != nil {
        handleJobError(dbService, err, msg, jobEntry)
        return
    }
case "slack":
    if err := processSlackJob(dbService, job, jobEntry, sTime); err != nil {
        handleJobError(dbService, err, msg, jobEntry)
        return
    }
case "webhook":
    if err := processWebhookJob(dbService, job, jobEntry, sTime); err != nil {
        handleJobError(dbService, err, msg, jobEntry)
        return
    }
default:
    handleJobError(dbService, fmt.Errorf("invalid event type: %s", job.Type), msg, jobEntry)
}

Then we switch-case on the job.Type and call further task process functions based on the job type.

If we get error from the task process functions, we call handleJobError() function and return.

func handleJobError(dbService *DBService, err error, msg *amqp091.Delivery, jobEntry *models.JobEntry) {
    log.Println("error in processing job: ", err.Error(), ", retries: ", jobEntry.Retries)
    time.Sleep(2 * time.Second)
    dbService.markJobAsFailed(err, jobEntry.Retries+1, jobEntry)
    msg.Nack(false, true)
}

In handleJobError() function, we sleep for 2 seconds before retrying.

We mark the job run entry as failed in database using markJobAsFailed() function. The job entry’s status will move to permanently_failed if it fails all 3 retries.

Then we negative acknowledge Nack() the message and pass requeue as true to put this message back into the queue.

On success processing of the job inside of the task processing function, we use markJobAsCompleted() function from dbService to update the job entry status to completed.

msg.Ack(false)

Then finally in end of processMessage() function, we Ack() the message.
In this way, the processing of job is done after it was consumed!

Conclusion

In this blog, we have learned about how we can schedule cron jobs and publish them to RabbitMQ’s queue. And how these messages can be consumed from the queue.

And we also learned, after consuming how we can acknowledge and retry on job failures.

This blog contained parts from the full project I created to also provide the dashboard for creating jobs, listing all jobs and view the job run statuses as running/completed/failed(for retries)/permanently_failed.

Make sure to give it a go:

If you find this article helpful, don't forget to hit the ❤️ button.

Check out my website here and feel free to connect.

Happy Coding! 👨‍💻

This article provides an optimal and modular project structure for backend projects in Golang, which I use as a starting template for my projects to ensure they are scalable and well maintained.

Along with the structure, it also includes a starter codebase built using gin framework, which can be used as a template.

Why Project Structure Matters?

Designing the structure of a project is important right from the beginning of the development. As the project grows, so does the codebase & complexity, which needs to be distributed properly across well-thought folders and files.

A clear structure makes it easy for the developers to navigate through the code as it grows and also lets future developers focus on implementing new features rather than spending time searching for relevant code.

Folder Structure

/api

The api folder consists of sub-folders to maintain the versioning of API to manage changes without breaking existing clients using previous versions.

├── api/
│   └── v1/
│       └── routes.go 
│       └── userRoutes.go

The sub-folders for versions named as v1, v2, etc contains routes.go files which basically registers different routes divided into different files based on entity/features.

// routes.go
package v1

import (
    "database/sql"

    "github.com/gin-gonic/gin"
)

func RegisterRoutes(router *gin.Engine, db *sql.DB) {
    registerUserRoutes(router, db)
}

// userRoutes.go
package v1

import (
    "database/sql"

    "github.com/Aniketyadav44/go-backend-template/internal/handlers"
    "github.com/Aniketyadav44/go-backend-template/internal/services"
    "github.com/gin-gonic/gin"
)

func registerUserRoutes(router *gin.Engine, db *sql.DB) {
    userService := services.NewUserService(db)
    userHandler := handlers.NewUserHandler(userService)

    v1 := router.Group("/api/v1")
    {
        v1.GET("/users", userHandler.GetAllUsers)
    }
}

Here, we basically created a service and handler instance by first injecting the database dependency in the service and then passing this service to the handler.

Then we defined a route group for /api/v1 path named as v1 and on this, we register our endpoint methods using the respective handler functions.

/cmd

The cmd folder is used as the entry point of our backend application, which contains sub-folders depending on the type of application we are building.

e.g cmd/api for our REST API or cmd/grpc for a grpc service or cmd/cli for an CLI application.

These sub-folders have their main.go file which is the exact entry point file of the project.

├── cmd/
│   └── api/
│       └── main.go

// main.go
package main

import (
    "log"

    v1 "github.com/Aniketyadav44/go-backend-template/api/v1"
    "github.com/Aniketyadav44/go-backend-template/internal/config"
    "github.com/gin-gonic/gin"
)

func main() {
    cfg, err := config.LoadConfig()
    if err != nil {
        log.Fatal("error in loading config: ", err)
    }

    router := gin.Default()
    v1.RegisterRoutes(router, cfg.DB)

    if err := router.Run(":" + cfg.Port); err != nil {
        log.Fatal("error in starting server: ", err)
    }
}

Here, we first load our config instance which provides the port and database instance for this demo.

Next we create a Gin router instance and then we register our API routes from the v1 package in api/v1 on this router.

Finally, we start the server using router.Run() binding it to the port we loaded from .env in config.

/internal

the internal folder consists of the application level logic further divided into sub folders depending on the configuration, control(presentation), business logic, data models and middlewares.

├── internal/
│   ├── config/
│   │   └── config.go
│   │   └── db.go              
│   ├── handlers/
│   │   └── userHandler.go              
│   ├── services/
│   │   └── userService.go              
│   ├── models/
│   │   └── userModel.go               
│   └── middleware/
│       └── authMiddleware.go

/internal/config

the config sub-folder in internal folder contains files for the config.go file which loads the environment variables and initializes our database instance.

// config.go
package config

import (
    "database/sql"
    "os"

    "github.com/joho/godotenv"
    _ "github.com/lib/pq"
)

type Config struct {
    Port string
    DB   *sql.DB
}

func LoadConfig() (*Config, error) {
    if err := godotenv.Load(); err != nil {
        return nil, err
    }

    port := getEnvKey("PORT", "")
    dbHost := getEnvKey("DB_HOST", "")
    dbPort := getEnvKey("DB_PORT", "")
    dbUser := getEnvKey("DB_USERNAME", "")
    dbPassword := getEnvKey("DB_PASSWORD", "")
    dbName := getEnvKey("DB_NAME", "")

    db, err := loadDb(dbHost, dbPort, dbUser, dbPassword, dbName)
    if err != nil {
        return nil, err
    }

    return &Config{
        Port: port,
        DB:   db,
    }, nil
}

func getEnvKey(key, defaultValue string) string {
    if val, exists := os.LookupEnv(key); exists {
        return val
    }
    return defaultValue
}

In this, we have defined a Config struct which holds the application level configurations. Specifically in this demo template, there is our database connection instance(*sql.DB) and server port.

Then, there is a function LoadConfig() which returns the loaded Config along with any error. This function first loads the environment using godotenv package and then loads database connection using loadDb() function defined in /config/db.go file.

In this db.go file, we create our database connection instance using sql package and test our connection using the db.Ping() function

// db.go
package config

import (
    "database/sql"
    "fmt"
)

func loadDb(dbHost, dbPort, dbUser, dbPassword, dbName string) (*sql.DB, error) {
    psConnStr := fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable", dbUser, dbPassword, dbHost, dbPort, dbName)

    db, err := sql.Open("postgres", psConnStr)
    if err != nil {
        return nil, err
    }

    if err := db.Ping(); err != nil {
        return nil, err
    }

    return db, nil
}

Just like db.go, it can contain configuration of different databases/services like redis, kafka, etc.

/internal/handlers

This folder consists of the HTTP handlers which defines how the requests will be processed.

// userHandler.go
package handlers

import (
    "net/http"

    "github.com/Aniketyadav44/go-backend-template/internal/services"
    "github.com/gin-gonic/gin"
)

type UserHandler struct {
    service *services.UserService
}

func NewUserHandler(service *services.UserService) *UserHandler {
    return &UserHandler{
        service: service,
    }
}

// ... handler functions defined here
func (h *UserHandler) GetAllUsers(c *gin.Context) {
    users, err := h.service.GetAllUsers()
    if err != nil {
        c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, gin.H{"users": users})
}

Here, we have defined a UserHandler struct which has a reference to it’s service. This service contains db connection injected which holds all of the business logic for specific handler functions to be called on specific routes.

Then, a constructor function NewUserHandler() is defined which basically initializes and returns a new handler with service dependency injected.

/internal/services

This folder consists of the services which holds all of the business logic for specific handlers.

// userService.go
package services

import (
    "database/sql"

    "github.com/Aniketyadav44/go-backend-template/internal/models"
)

type UserService struct {
    db *sql.DB
}

func NewUserService(db *sql.DB) *UserService {
    return &UserService{
        db: db,
    }
}

func (s *UserService) GetAllUsers() ([]models.User, error) {
    // ... get users logic from database connection -> s.db
}

Here, we have defined UserService struct that holds a reference to the database connection (*sql.DB).

Then, there is a constructor function NewUserService() which initializes and returns new instance of service with the database connection injected.

/internal/models

This folder consists of the different data structures used across our application. This also contains the structures for our request body of POST APIs

// userModel.go example
package models

type User struct {
    ID        string    `json:"id"`
    Username  string    `json:"username"`
    Email     string    `json:"email"`
}

/internal/middlewares

This folder consists of the different middlewares used for our APIs divided into different files.

// authMiddleware.go
package middlewares

import (
    "fmt"
    "net/http"

    "github.com/gin-gonic/gin"
)

func AuthMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        // ...logic
        c.Next()
    }
}

/pkg

This folder consists of various sub-folders such as responses, utils, etc which holds all of the re-usable code to be imported across our application.

for e.g.

• /pkg/utils/utils.go can consist of common utilities functions

• /pkg/utils/crypto.go can consist of the encryption, decryption, hashing, etc functions

• /pkg/responses/errorResponses.go can consist of different error responses to be sent in error conditions.

Conclusion

Designing a clean and modular structure is very important for a scalable and maintainable codebase. By organizing code files into different logical layers, we make our project easier to understand and extend.

This gin based template can be used to quickly get started with a new project.

This template can be found here:

If you find this article helpful, don't forget to hit the ❤️ button.

Check out my website here and feel free to connect.

Happy Coding! 👨‍💻

Golang has made it very simple to apply concurrency in our programs and worker pool is one of the amazing concurrency pattern that we can simply create with go.

Definition

The technical definition is: (Yes, I copied this from internet 🙂)

A worker pool is a software design pattern where a set of worker threads or processes (the "pool") are created to concurrently execute tasks from a queue

In simple terms of golang, worker pool allows us to run multiple tasks(jobs) concurrently using a fixed number of goroutines(workers) that pulls tasks from a shared channel(acting as queue) without need of creating a new goroutine for each task. Thus efficiently running tasks with less resource usage.

Why Worker Pools?

Worker pool makes it possible to run a huge number of tasks on a limited hardware resource.

For e.g. In case of hitting GET requests concurrently to 100 urls, we can directly run a goroutine for each GET call. It’s simple in golang.

package main

import (
    "fmt"
    "net/http"
    "sync"
    "time"
)

type Result struct {
    Response *http.Response
    Error    error
}

func request(client *http.Client, url string, results chan<- Result, wg *sync.WaitGroup) {
    defer wg.Done()
    res, err := client.Get(url)
    results <- Result{res, err}
}

func main() {
    urls := []string{
        // 100 urls from https://gist.github.com/demersdesigns/4442cd84c1cc6c5ccda9b19eac1ba52b
    }
    results := make(chan Result, len(urls))
    wg := sync.WaitGroup{}
    client := http.Client{
        Timeout: 5 * time.Second,
    }

    wg.Add(len(urls))
    // starting goroutine for each url
    for _, v := range urls {
        go request(&client, v, results, &wg)
    }

    // collecting and printing results
    for i := 0; i < len(urls); i++ {
        res := <-results
        if res.Error != nil {
            fmt.Printf("error in request: %s\n", res.Error.Error())
        } else {
            fmt.Printf("request success for %s\n", res.Response.Request.URL)
        }
    }
    wg.Wait()
}

Here, we have created a urls list that holds 100 url string. Then we make a channel of Result struct that holds the response and error. Then we create a WaitGroup wg to wait for completion of goroutines and an http client with 5 seconds timeout. then we started 100 goroutines using range loop using the request function.

Inside this function, we first defer wg.Done() and hit the GET request. Then the Result is sent to the results channel.

Simultaneously in main program, we loop for urls length and get result from results channel and print them.

This program is fast at first look and also very simple, but if we want to do similar thing on let’s say 1000, 10,000 or even 100,000 URLs? it can easily max out the system resources for a huge dataset at scale.

But it can be improved. This is where, worker pools can help!

Worker Pool

Worker pool allows us to run fixed number of goroutines, where each goroutine can call multiple GET urls from the shared queue channel.

package main

import (
    "fmt"
    "net/http"
    "sync"
    "time"
)

type Job struct {
    URL string
}

type JobResult struct {
    Response *http.Response
    Error    error
}

func workers(client *http.Client, jobs <-chan Job, results chan<- JobResult, wg *sync.WaitGroup) {
    for job := range jobs {
        res, err := client.Get(job.URL)
        results <- JobResult{res, err}
        wg.Done()
    }
}

func main() {
    urls := []string{
        // 100 urls from https://gist.github.com/demersdesigns/4442cd84c1cc6c5ccda9b19eac1ba52b
    }
    numWorkers := 10
    client := http.Client{
        Timeout: 5 * time.Second,
    }

    jobs := make(chan Job, len(urls))
    results := make(chan JobResult, len(urls))
    wg := sync.WaitGroup{}

    // starting a fixed number of worker goroutines
    for i := 0; i < numWorkers; i++ {
        go workers(&client, jobs, results, &wg)
    }

    wg.Add(len(urls))
    // sending jobs to the job queue
    for i := 0; i < len(urls); i++ {
        jobs <- Job{urls[i]}
    }
    close(jobs)

    // collecting and printing results
    for i := 0; i < len(urls); i++ {
        res := <-results
        if res.Error != nil {
            fmt.Println("error: ", res.Error.Error())
        } else {
            fmt.Println(res.Response.Request.URL)
        }
    }
    wg.Wait()
}

In this,

we have created a new struct Job to demonstrate a task/job that holds the url.

We used a buffered channel jobs of Job with numWorkers capacity that specifies total workers we want to deploy. This channel will be our shared queue from which the worker goroutines will pick their task.

Then we deploy exactly numWorkers number of workers as goroutines using workers function. Inside this function, we use range over the jobs channel that picks tasks from this channel. On picking a task, it makes the GET call, puts the Result in results channel and completes with wg.Done()

Then, we put counter to our WaitGroup and start putting jobs to our jobs channel by looping over urls list. The we close the channel using close(jobs) to signal the workers that there are no more tasks and they can exit the range loop

After this, we get results using a loop from results channel and print to the console.

Here, inside the workers function, we loop over the jobs channel and pick the available job/task in this channel. As the workers goes on picking a task from this channel, it keeps on reducing and eventually all of the tasks gets picked and completed. This works like when worker1 pick task1, worker2 picks task2. Then worker1 can go to pick task3 and thus each worker can eventually execute multiple tasks.

Conclusion

So, instead of executing 1 task per goroutine, we can execute multiple tasks from a single goroutine. This is the main purpose of using a worker pool!

In the given examples, we are running total of 10 worker goroutines to call 100 GET calls using a worker pool, which is much efficient than previous example that spins 100 goroutines for same data.

If you find this article helpful, don't forget to hit the ❤️ button.

Check out my website here and feel free to connect.

Happy Coding! 👨‍💻

Hello DEV fam, today I will be showing you how to create a basic REST API in Go language using Gorilla mux package with very simple steps, so even if your are very new to this amazing language you can follow this tutorial without any hesitation and problems. So now let's get right into it!

What are we doing today?

We will be creating a very very basic REST Api on our localhost to serve various GET POST PUT DELETE requests.
As this is going to be a beginner's tutorial to Gorilla Mux package, so we will not be looking into any of the database queries. We are going to use a sample data collection to assume it as a data coming from some database.
So without any delays let's get started.

Any Pre-Requisites?

But wait wait wait... You may ask what should I know in advance to follow this tutorial?
Hmm... Nothing fancy, just basics of Go language and that's it!
Else I will be explaining you about using & designing basic Crud REST API.

Let's see, what we will have in the end

We are going to create a simple application where,
Getting list of users in response to valid GET request at /users
Getting an user in response to valid GET request at /user/{id}
Creating an user in response to valid POST request at /user
Updating an user in response to valid PUT request at /user/{id}
Deleting an user in response to valid DELETE request at /user/{id}

The first step

Initialize your project folder by creating a folder where we will be writing our code. So begin with following commands in your terminal.
First to create the folder and another to move into that folder

mkdir go-mux-tut
cd go-mux-tut

Now initialize the GO modules using a github repo follow up address

go mod init github.com/user_name/repo_name

Now, it's time to fetch the required Gorilla Mux module into our project using following command
mux package - The Gorilla Mux router (also known as "HTTP request multiplexer")

go get -u github.com/gorilla/mux

Now designing the Project structure

create main.go app.go model.go controller.go in the project folder
So our project structure now looks something like this:

-app.go
-controller.go
-model.go
-main.go
-go.mod
-go.sum

Now the basic setup of project is completed, let's get started with coding!

Creating a router

First starting with app.go file, create a new router variable Router using mux.NewRouter() from the mux package, for this import "github.com/gorilla/mux"

var Router = mux.NewRouter()

Now let's create a function HandleRoutes() to handle different routes serving on this Router

func HandleRoutes() {
    Router.HandleFunc("/users", GetAllUsers).Methods("GET")
    Router.HandleFunc("/user/{id}", GetUser).Methods("GET")
    Router.HandleFunc("/user", CreateUser).Methods("POST")
    Router.HandleFunc("/user/{id}", UpdateUser).Methods("PUT")
    Router.HandleFunc("/user/{id}", DeleteUser).Methods("DELETE")
}

Here using Router from the mux package, we create different routes using HandleFunc() function which takes string path and a function of http.ResponseWriter, *http.Response called controllers
We defined:

• /users path and assigned it with GetAllUsers controller, with method GET to get all the users
• /user/{id} path and assigned it with GetUser controller by passing id in path, with method GET to get user with specified id
• /user path and assigned it with CreateUser controller, with method POST to create a new user. In this we will be passing new user's data in request body
• /user/{id} path and assigned it with UpdateUser controller, with method PUT to update an existing user and we will be passing the updated data in request body
• /users/{id} path and assigned it with DeleteUser controller, with method DELETE to delete an user

The final app.go file looks like this

package main

import (
    "github.com/gorilla/mux"
)

var Router = mux.NewRouter()

func HandleRoutes() {
    Router.HandleFunc("/users", GetAllUsers).Methods("GET")
    Router.HandleFunc("/user/{id}", GetUser).Methods("GET")
    Router.HandleFunc("/user", CreateUser).Methods("POST")
    Router.HandleFunc("/user/{id}", UpdateUser).Methods("PUT")
    Router.HandleFunc("/user/{id}", DeleteUser).Methods("DELETE")
}

Creating data model

To work with Request and serving Responses, we need to first have the data.
For this we will first create the User struct

type User struct {
    ID        int    `json:"id"`
    FirstName string `json:"firstName"`
    LastName  string `json:"lastName"`
}

This User struct has an ID of type int with it's json name as id, FirstName of type string with it's json name as firstName, LastName of type string with it's json name as lastName

Now we will create a sample slice of users to work with

var UsersData = []User{
    User{1, "Stephanie", "Turner"},
    User{2, "Anna", "Edmunds"},
    User{3, "Jan", "Vaughan"},
    User{4, "Grace", "North"},
    User{5, "Piers", "Morrison"},
}

The final model.go file looks like this

package main

type User struct {
    ID        int    `json:"id"`
    FirstName string `json:"firstName"`
    LastName  string `json:"lastName"`
}

var UsersData = []User{
    User{1, "Stephanie", "Turner"},
    User{2, "Anna", "Edmunds"},
    User{3, "Jan", "Vaughan"},
    User{4, "Grace", "North"},
    User{5, "Piers", "Morrison"},
}

Creating the controllers

Now comes the important part, i.e creating the controllers which are the functions responsible for serving different paths of our simple API.

We're first required to import some packages, which are encoding/json, math/rand, net/http, strconv, github.com/gorilla/mux
Usage of each package will be explained further

import (
    "encoding/json"
    "math/rand"
    "net/http"
    "strconv"

    "github.com/gorilla/mux"
)

Starting with the simplest, GetAllUsers Function to return all users

func GetAllUsers(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(UsersData)
}

Let's understand each component of this function,
we have two parameters here, w http.ResponseWriter which has a Write method which accepts a byte slice and writes the data to the connection as part of an HTTP response and r *http.Request which deals with the request stuff.
We then define the type of Data we are sending using the Content-Type parameter in response's header and setting it as application/json to send json type data
Then finally we send the response by encoding the data to json format

Now creating GetUser Function to send an user with specific id

func GetUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")

    vars := mux.Vars(r)
    paramId, _ := strconv.Atoi(vars["id"])

    for _, user := range UsersData {
        if user.ID == paramId {
            json.NewEncoder(w).Encode(user)
            return
        }
    }
}

In this, we first get the map[string]string of parameters from the request r using mux.Vars(r) As in our case, we need the id as int type, so we then get the id from vars map and then convert it to int type using strconv package.
Then we run a loop over our UsersData and find the user with this id, then return it in response.

Working with CreateUser Function to create a new user

func CreateUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Conent-Type", "application/json")

    var user User
    user.ID = rand.Intn(10000)

    json.NewDecoder(r.Body).Decode(&user)
    UsersData = append(UsersData, user)

    msg := "User created successfully"
    json.NewEncoder(w).Encode(ResponseMsg{msg, user})
}

In this, we first created a new user and assigned it with a random id within a range of 10000 using math/rand package.
Then we access the data we got in request body and decode it from json format to our User struct type and store it in user variable. Then finally we store this received user into our UsersData slice. To send a response also with message, we have created another struct called ResponseMsg , which has a message and user data to send

type ResponseMsg struct {
    Msg  string `json:"message"`
    User User   `json:"user"`
}

Then we send this response with a message of "User created successfully" and created user's data

Deleting user with DeleteUser Function

func DeleteUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Conent-Type", "application/json")

    vars := mux.Vars(r)
    paramId, _ := strconv.Atoi(vars["id"])

    for index, user := range UsersData {
        if user.ID == paramId {
            UsersData = append(UsersData[:index], UsersData[index+1:]...)
        }
    }

    msg := "User deleted successfully"
    json.NewEncoder(w).Encode(msg)
}

In this we have mentioned the header's Content-Type and got the id from requested path as we did earlier.
Then we ran a loop over the User's data to find the required user by comparing user's id with passed id.
When we get the required user, now we need to remove i.e delete this user from our UsersData slice.
To do so, we apply slicing using append() function by first cutting the original slice from 0th index upto that user's index , and then combining it with rest of the slice from user's index+1th position until last position.
Then finally we send a response with a message of "User deleted successfully"

Working with UpdateUser Function, to update an user

func UpdateUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Conent-Type", "application/json")

    vars := mux.Vars(r)
    paramId, _ := strconv.Atoi(vars["id"])

    for index, user := range UsersData {
        if user.ID == paramId {
            UsersData = append(UsersData[:index], UsersData[index+1:]...)
        }
    }

    var user User
    user.ID = paramId
    json.NewDecoder(r.Body).Decode(&user)
    UsersData = append(UsersData, user)

    msg := "User updated successfully"
    json.NewEncoder(w).Encode(ResponseMsg{msg, user})
}

This has a little complex logic... No worries, let's understand it line by line.
So first we defined the header and got the id from request path as we have done earlier.
After that we ran a loop over the UsersData to find the user with that id, and here we first delete this user using above DeleteUser's logic.
Now the old id's user is deleted, we then create a new user with that same id and then add it to the UsersData slice.
In this manner, we have successfully updated the user.
We then send a response with a message saying "User updated successfully" along with the updated user's data.

The final controller.go file look like this

package main

import (
    "encoding/json"
    "math/rand"
    "net/http"
    "strconv"

    "github.com/gorilla/mux"
)

func GetAllUsers(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(UsersData)
}

func GetUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")

    vars := mux.Vars(r)
    paramId, _ := strconv.Atoi(vars["id"])

    for _, user := range UsersData {
        if user.ID == paramId {
            json.NewEncoder(w).Encode(user)
            return
        }
    }
}

type ResponseMsg struct {
    Msg  string `json:"message"`
    User User   `json:"user"`
}

func CreateUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Conent-Type", "application/json")

    var user User
    user.ID = rand.Intn(10000)

    json.NewDecoder(r.Body).Decode(&user)
    UsersData = append(UsersData, user)

    msg := "User created successfully"
    json.NewEncoder(w).Encode(ResponseMsg{msg, user})
}

func UpdateUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Conent-Type", "application/json")

    vars := mux.Vars(r)
    paramId, _ := strconv.Atoi(vars["id"])

    for index, user := range UsersData {
        if user.ID == paramId {
            UsersData = append(UsersData[:index], UsersData[index+1:]...)
        }
    }

    var user User
    user.ID = paramId
    json.NewDecoder(r.Body).Decode(&user)
    UsersData = append(UsersData, user)

    msg := "User updated successfully"
    json.NewEncoder(w).Encode(ResponseMsg{msg, user})
}

func DeleteUser(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Conent-Type", "application/json")

    vars := mux.Vars(r)
    paramId, _ := strconv.Atoi(vars["id"])

    for index, user := range UsersData {
        if user.ID == paramId {
            UsersData = append(UsersData[:index], UsersData[index+1:]...)
        }
    }

    msg := "User deleted successfully"
    json.NewEncoder(w).Encode(msg)
}

Now we have only created different components of our API, now it's time to connect all the components in main.go file

Connecting all the components and starting our API

Now we need to get the Router from app.go file and call HandleRoutes() function.

So first let's import the required packages which are fmt, log, net/http

import (
    "fmt"
    "log"
    "net/http"
)

To start the server, we need to call the ListenAndServe function from net/http package by passing the port address i.e 8000 in our case and the router which we have created in app.go file

fmt.Println("Server started on port:8000")

log.Fatal(http.ListenAndServe(":8000", r))

The main.go file looks like this

package main

import (
    "fmt"
    "log"
    "net/http"
)

func main() {
    r := Router

    HandleRoutes()

    fmt.Println("Server started on port:8000")

    log.Fatal(http.ListenAndServe(":8000", r))
}

Finally here we are, done with creating the very basic API using Gorilla Mux in Go.
Now it's time to check our API using POSTMAN.

Testing our API using Postman

To test our api we need to run our main.go file, so enter following command in terminal while being in the project folder

go run .

You will get output something like this

Server started on port:8000

Now your api server has been started on localhost at port 8000 and this can be accessed at "http://localhost:8000"

Testing on Postman

• Get all Users
  To get all the users, in Postman enter the url "localhost:8000/users" and keeping the methods as GET
  We get the following response:

• Get specific User
  To get a specific user, in Postman enter the url "localhost:8000/user/3" to get the user with id 3. You can user another id in place of 3.
  We get the following response:

• Create User
  To create new user, in Postman enter the url "localhost:8000/user", then open Header tab and set Content-Type to application/json

Then create the request body from Body tab

Now set the method type to POST and hit Send,
Then we get a response somthing like this:

And when you will again run get all user request, you can see the changes

• Update User
  To update user, in Postman enter the url "localhost:8000/user/1" and mention the Header as done above.
  Now mention the updating information in request body using Body tab:

Now set the method as PUT and hit Send
So the response will look like:

• Deleting User
  To delete user, in Postman enter the url "localhost:8000/user/1" by keeping the method as DELETE You will get following response:

What we did today - Conclusion

Gorilla mux is an amazing package to work with and build APIs easily and efficiently. This was a simple example using a sample slice data. But we can use different databases like Relational SQL databases like MySQL, PostgreSQL, etc or NoSQLs like MongoDB, Redis,etc or Cloud databases like dynamoDB, etc.
Relax, no need to worry after listening all these advance techs. I will be making detailed tutorials on each just like this so stay tuned and Subscribe to our newsletter and never miss any upcoming articles and Follow me for more.

Here is the link for the Github repository of this project:
%[https://github.com/Aniketyadav44/go-mux-tut]

Let's Connect

LinkedIn Github Website Twitter Instagram

Follow me for such more detailed tutorials and articles. I try to make learning, programming and development in much fun and easier way possible.

So Keep Learning, and Enjoy making Technologies!

Peace!