Multiplexer: HTTP-Server & URL Crawler
Приложение представляет собой http-сервер с одним хендлером.
Хендлер на вход получает POST-запрос со списком url в json-формате.
Сервер запрашивает данные по всем этим url и возвращает результат клиенту в json-формате.
Если в процессе обработки хотя бы одного из url получена ошибка, обработка всего списка прекращается и клиенту возвращается текстовая ошибка.
Ограничения:
- для реализации задачи следует использовать Go 1.13 или выше
- использовать можно только компоненты стандартной библиотеки Go
- сервер не принимает запрос если количество url в в нем больше 20
- сервер не обслуживает больше чем 100 одновременных входящих http-запросов
- для каждого входящего запроса должно быть не больше 4 одновременных исходящих
- таймаут на запрос одного url - секунда
- обработка запроса может быть отменена клиентом в любой момент, это должно повлечь за собой остановку всех операций связанных с этим запросом
- сервис должен поддерживать 'graceful shutdown'
Run a Server
$ go run ./cmd/multiplexer/main.go
> 2021/10/28 17:21:24 app started on port: 80
Request Validation
POST-Method
$ curl http://localhost/crawler
> method not allowed: expected "POST": got "GET"
JSON Input
$ curl -X POST http://localhost/crawler
> unsupported "Content-Type" header: expected "application/json": got ""
Empty Body
bad request: empty request body ">
$ curl -X POST http://localhost/crawler -H "Content-Type: application/json" > bad request: empty request body
Input Contract Compliance
bad request: invalid character 's' looking for beginning of value ">
$ curl -X POST http://localhost/crawler -d "some random data" \ -H "Content-Type: application/json" > bad request: invalid character 's' looking for beginning of value
bad request: no URLs passed ">
$ curl -X POST http://localhost/crawler -d '{"some":"field"}' \ -H "Content-Type: application/json" > bad request: no URLs passed
max number of URLs exceeded: 22 of 20" ">
$ curl -X POST http://localhost/crawler \
-H "Content-Type: application/json" \
-d '{"urls":[
"1", "2", "3", "4", "5", "6", "7", "8", "9", "10",
"11", "12", "13", "14", "15", "16", "17", "18", "19",
"20", "21", "22"]}'
> max number of URLs exceeded: 22 of 20"
invalid url: "some random text" ">
$ curl -X POST http://localhost/crawler \
-H "Content-Type: application/json" \
-d '{"urls":["some random text"]}'
> invalid url: "some random text"
Error Handling
HTTP Status Code Check
failed to crawl "https://httpstat.us/500": unexpected response status code: 500 ">
$ curl -X POST http://localhost/crawler \
-H "Content-Type: application/json" \
-d '{"urls":["https://httpstat.us/500"]}'
> failed to crawl "https://httpstat.us/500": unexpected response status code: 500
Request Timeout
failed to crawl "https://httpstat.us/200?sleep=5000": failed to send a request: Get "https://httpstat.us/200?sleep=5000": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ">
$ curl -X POST http://localhost/crawler \
-H "Content-Type: application/json" \
-d '{"urls":["https://httpstat.us/200?sleep=5000"]}'
> failed to crawl "https://httpstat.us/200?sleep=5000":
failed to send a request: Get "https://httpstat.us/200?sleep=5000":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Exit Fast & Context Cancel
2021/10/28 16:53:13 app started on port: 80
2021/10/28 16:53:19 crawler: received 3 tasks: validating URL format
2021/10/28 16:53:19 crawler: starting 3 workers
2021/10/28 16:53:19 crawler: sending request: http://google.com
2021/10/28 16:53:19 crawler: sending request: http://yandex.ru
2021/10/28 16:53:19 crawler: sending request: http://69.63.176.13
2021/10/28 16:53:19 crawler: worker stopped: no more tasks
2021/10/28 16:53:19 crawler: unmarshal response body to JSON: invalid character '<' looking for beginning of value
2021/10/28 16:53:19 crawler: worker stopped: no more tasks
2021/10/28 16:53:19 crawler: error occurred: stopping other goroutines
2021/10/28 16:53:19 crawler: send request: Get "http://69.63.176.13": context canceled
2021/10/28 16:53:19 crawler: worker stopped: context canceled
2021/10/28 16:53:19 crawler: send request: Get "http://www.google.com/": context canceled
2021/10/28 16:53:19 crawler: error occurred: skipping new results
2021/10/28 16:53:19 crawler: error occurred: skipping new results
2021/10/28 16:53:19 crawler: worker stopped: context canceled
2021/10/28 16:53:19 crawler: results channel closed
2021/10/28 16:53:19 crawler: exit with error: failed to crawl "http://yandex.ru": unmarshal response body to JSON: invalid character '<' looking for beginning of value
2021/10/28 16:53:19 handler: failed to crawl "http://yandex.ru": unmarshal response body to JSON: invalid character '<' looking for beginning of value
Graceful Shutdown
2021/10/28 16:53:13 app started on port: 80
...
^C2021/10/28 16:56:44 OS signal received: interrupt
2021/10/28 16:56:44 http: setting graceful timeout: 3.00s
2021/10/28 16:56:44 http: awaiting traffic to stop: 3.00s
2021/10/28 16:56:44 http: shutting down: disabling keep-alive
2021/10/28 16:56:44 closer: http: shutting down: context deadline exceeded
Process finished with exit code 0
Limited Number of Simultaneous Incoming Requests
The problem is solved with a simple buffered-channel window. Before new connection can be established, it has to acquire a lock (get queued to a channel). When connection is closed a lock is released.
Limited Number of Outgoing Requests
The problem can be solved in multiple ways, e.g. having a fixed number of workers that evenly pull tasks from the queue, or we can just run a goroutine per URL and try to acquire a spot in the buffered-channel window to limit the number of concurrently running tasks.
An unlimited number of goroutines (due to unknown size of the URLs array) can end up with waste of resources and crash afterwards. That's why I chose a worker-pool solution, it solves this exact problem just fine.
Happy Path
{ "results": [ { "url": "https://jsonplaceholder.typicode.com/todos/3", "response": { "code": 200, "body": { "userId": 1, "id": 3, "title": "fugiat veniam minus", "completed": false } } }, ... { "url": "https://jsonplaceholder.typicode.com/todos/20", "response": { "code": 200, "body": { "userId": 1, "id": 20, "title": "ullam nobis libero sapiente ad optio sint", "completed": true } } } ] } ">
$ curl -X POST http://localhost/crawler -H "Content-Type: application/json" -d '{"urls":[ "https://jsonplaceholder.typicode.com/todos/1", "https://jsonplaceholder.typicode.com/todos/2", "https://jsonplaceholder.typicode.com/todos/3", "https://jsonplaceholder.typicode.com/todos/4", "https://jsonplaceholder.typicode.com/todos/5", "https://jsonplaceholder.typicode.com/todos/6", "https://jsonplaceholder.typicode.com/todos/7", "https://jsonplaceholder.typicode.com/todos/8", "https://jsonplaceholder.typicode.com/todos/9", "https://jsonplaceholder.typicode.com/todos/10", "https://jsonplaceholder.typicode.com/todos/11", "https://jsonplaceholder.typicode.com/todos/12", "https://jsonplaceholder.typicode.com/todos/13", "https://jsonplaceholder.typicode.com/todos/14", "https://jsonplaceholder.typicode.com/todos/15", "https://jsonplaceholder.typicode.com/todos/16", "https://jsonplaceholder.typicode.com/todos/17", "https://jsonplaceholder.typicode.com/todos/18", "https://jsonplaceholder.typicode.com/todos/19", "https://jsonplaceholder.typicode.com/todos/20" ]}' > { "results": [ { "url": "https://jsonplaceholder.typicode.com/todos/3", "response": { "code": 200, "body": { "userId": 1, "id": 3, "title": "fugiat veniam minus", "completed": false } } }, ... { "url": "https://jsonplaceholder.typicode.com/todos/20", "response": { "code": 200, "body": { "userId": 1, "id": 20, "title": "ullam nobis libero sapiente ad optio sint", "completed": true } } } ] }
