I have a text-generator language model that is compressed and is in .bin format, and it can accessed by command line terminal. It generates about one word per second and prints out every word in realtime when accessed through command line. I would like to deploy my model using cortex but I'm struggling to get the output in realtime word-by-word. Right now my code prints out only one line a time.

import subprocess
def run_command(text):
    command = ['./mycommand', text]
    process = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True, bufsize=-1)
    while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output:
            print(output.strip())

run_command('TEXT')

One line may include about 20 words, so that means 1 line will be displayed every 20 seconds (since my model outputs roughly 1 word/ second). I would really like the output to be more dynamic and output one word at a time (as it does through command line) instead of just one line. Is there a way this can be achieved?

I would like to use Cortex functionality, to create an application where each user will be able to request and communicate with AWS instance for a period of time. In this scenario, data of each user will be processed and stored on one whole AWS instance. From the documentation, I understand that each API call will use an instance that it is not busy at the moment. It wouldn’t be ideal if by making an API call, a user would receive sensitive data stored by a another user on the same instance. Would it be possible to somehow mark an instance to which an API call is being made? That way the data of individual users wouldn’t be made accesible to everyone, but only to those users who request/use an instance.

I'm trying to send audio files, which are fairly large, to the server and am getting a resource exhausted error. Is there any way to configure the server in order to increase the maximum allowed message size?

Here's the stack trace:

2020-12-24 23:30:14.941839:cortex:pid-2247:INFO:500 Internal Server Error POST /
2020-12-24 23:30:14.942071:cortex:pid-2247:ERROR:Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/envs/env/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line
390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/fastapi/applications.py", line 181, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/cortex_internal/serve/serve.py", line 187, in parse_payload
    return await call_next(request)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/cortex_internal/serve/serve.py", line 134, in register_request
    response = await call_next(request)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
 File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/fastapi/routing.py", line 183, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/opt/conda/envs/env/lib/python3.6/site-packages/fastapi/routing.py", line 135, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/opt/conda/envs/env/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/cortex_internal/serve/serve.py", line 200, in predict
    prediction = predictor_impl.predict(**kwargs)
  File "/mnt/project/serving/cortex_server.py", line 10, in predict
    return self.client.predict({"waveform": np.array(payload["audio"]).astype("float32")})
  File "/opt/conda/envs/env/lib/python3.6/site-packages/cortex_internal/lib/client/tensorflow.py", line
114, in predict
    return self._run_inference(model_input, consts.SINGLE_MODEL_NAME, model_version)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/cortex_internal/lib/client/tensorflow.py", line
164, in _run_inference
    return self._client.predict(model_input, model_name, model_version)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/cortex_internal/lib/model/tfs.py", line 376, in
predict
    response_proto = self._pred.Predict(prediction_request, timeout=timeout)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/conda/envs/env/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.RESOURCE_EXHAUSTED
        details = "Received message larger than max (102484524 vs. 4194304)"
        debug_error_string = "{"created":"@1608852614.937822193","description":"Received message larger
than max (102484524 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":203,"grpc_status":8}"

As you have mentioned in docs gpus.md about the limiting the gpu ram, I know what exactly code snippet I have to use, but dont know exactly where I have to write that code snippet in cortex source code.

mem_limit_mb = 1024 for gpu in tf.config.list_physical_devices("GPU"): tf.config.set_logical_device_configuration( gpu, [tf.config.LogicalDeviceConfiguration(memory_limit=mem_limit_mb)])

Description

In order to authenticate with the Cortex operator, the Cortex CLI should be able to use aws_session_token (currently only static credentials are supported).

Also, consider enabling auth via IAM role (e.g. inherited from Lambda, EC2)

I'm trying to create a microservice to manage my cluster via Cortex and Lambda. AWS Lambda requires python dependencies to be packaged and uploaded as a .zip files. How can I package Cortex library to .zip?

I have website which runs on https:// and I can't make Cortex API XmlHttpRequest requests over it.

When running on localhost using http://, everything works fine:

function postData(url = '', data = {}) {
  // Default options are marked with *
  const response = await fetch(url, {
    method: 'POST', // *GET, POST, PUT, DELETE, etc.
    mode: 'cors', // no-cors, *cors, same-origin
    cache: 'no-cache', // *default, no-cache, reload, force-cache, only-if-cached
    credentials: 'same-origin', // include, *same-origin, omit
    headers: {
      'Content-Type': 'application/json'
      // 'Content-Type': 'application/x-www-form-urlencoded',
    },
    redirect: 'follow', // manual, *follow, error
    referrerPolicy: 'no-referrer', // no-referrer, *no-referrer-when-downgrade, origin, origin-when-cross-origin, same-origin, strict-origin, strict-origin-when-cross-origin, unsafe-url
    body: JSON.stringify(data) // body data type must match "Content-Type" header
  });
  return response.json(); // parses JSON response into native JavaScript objects
}

But when making the same request on a https:// page gives following:

Mixed Content: The page at 'https://www.@' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://a6cc8d4dee22a448e81bb29862332bf0-93580d7c9d7d2256.elb.us-east-2.amazonaws.com/newtest-user'. This request has been blocked; the content must be served over HTTPS.

How can I access Cortex API over HTTPS?

Version

cli version: 0.18.1

Description

Intermittent 503 errors on AWS cluster.

Configuration

cortex.yaml

# cortex.yaml

- name: offer-features
  predictor:
    type: python
    path: predictor.py
    config:
      bucket: XXXXXXXXXXXXXXXXXXXX
  compute:
    cpu: 1  # CPU request per replica, e.g. 200m or 1 (200m is equivalent to 0.2) (default: 200m)
    gpu: 0  # GPU request per replica (default: 0)
    inf: 0 # Inferentia ASIC request per replica (default: 0)
    mem: 1Gi
  autoscaling:
    min_replicas: 2
    max_replicas: 3
    init_replicas: 2
    max_replica_concurrency: 13
    target_replica_concurrency: 5
    window: 1m0s
    downscale_stabilization_period: 5m0s
    upscale_stabilization_period: 1m0s
    max_downscale_factor: 0.75
    max_upscale_factor: 1.5
    downscale_tolerance: 0.05
    upscale_tolerance: 0.05

# cluster.yaml

# AWS credentials (if not specified, ~/.aws/credentials will be checked) (can be overridden by $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY)
aws_access_key_id: XXXXXXXXXXXXXX
aws_secret_access_key: XXXXXXXXXXXXXXXXX

# optional AWS credentials for the operator which may be used to restrict its AWS access (defaults to the AWS credentials set above)
cortex_aws_access_key_id: XXXXXXXXXXXXXXXX
cortex_aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXX

# EKS cluster name for cortex (default: cortex)
cluster_name: cortex

# AWS region
region: us-east-1

# S3 bucket (default: <cluster_name>-<RANDOM_ID>)
# note: your cortex cluster uses this bucket for metadata storage, and it should not be accessed directly (a separate bucket should be used for your models)
bucket: # cortex-<RANDOM_ID>

# list of availability zones for your region (default: 3 random availability zones from the specified region)
availability_zones: # e.g. [us-east-1a, us-east-1b, us-east-1c]

# instance type
instance_type: t3.medium

# minimum number of instances (must be >= 0)
min_instances: 1

# maximum number of instances (must be >= 1)
max_instances: 5

# disk storage size per instance (GB) (default: 50)
instance_volume_size: 50

# instance volume type [gp2, io1, st1, sc1] (default: gp2)
instance_volume_type: gp2

# instance volume iops (only applicable to io1 storage type) (default: 3000)
# instance_volume_iops: 3000

# whether the subnets used for EC2 instances should be public or private (default: "public")
# if "public", instances will be assigned public IP addresses; if "private", instances won't have public IPs and a NAT gateway will be created to allow outgoing network requests
# see https://docs.cortex.dev/v/0.18/miscellaneous/security#private-cluster for more information
subnet_visibility: public  # must be "public" or "private"

# whether to include a NAT gateway with the cluster (a NAT gateway is necessary when using private subnets)
# default value is "none" if subnet_visibility is set to "public"; "single" if subnet_visibility is "private"
nat_gateway: none  # must be "none", "single", or "highly_available" (highly_available means one NAT gateway per availability zone)

# whether the API load balancer should be internet-facing or internal (default: "internet-facing")
# note: if using "internal", APIs will still be accessible via the public API Gateway endpoint unless you also disable API Gateway in your API's configuration (if you do that, you must configure VPC Peering to connect to your APIs)
# see https://docs.cortex.dev/v/0.18/miscellaneous/security#private-cluster for more information
api_load_balancer_scheme: internet-facing  # must be "internet-facing" or "internal"

# whether the operator load balancer should be internet-facing or internal (default: "internet-facing")
# note: if using "internal", you must configure VPC Peering to connect your CLI to your cluster operator (https://docs.cortex.dev/v/0.18/guides/vpc-peering)
# see https://docs.cortex.dev/v/0.18/miscellaneous/security#private-cluster for more information
operator_load_balancer_scheme: internet-facing  # must be "internet-facing" or "internal"

# CloudWatch log group for cortex (default: <cluster_name>)
log_group: cortex

# additional tags to assign to aws resources for labelling and cost allocation (by default, all resources will be tagged with cortex.dev/cluster-name=<cluster_name>)
tags:  # <string>: <string> map of key/value pairs

# whether to use spot instances in the cluster (default: false)
# see https://docs.cortex.dev/v/0.18/cluster-management/spot-instances for additional details on spot configuration
spot: false

# see https://docs.cortex.dev/v/0.18/guides/custom-domain for instructions on how to set up a custom domain
ssl_certificate_arn: XXXXXXXXXXXXXXXXXXXXXXXXXXXX

Steps to reproduce

• Spin up instances on AWS.
• Wait a couple of days / hours (varies).
• Notice sudden 503 errors

Expected behavior

It should work

Actual behavior

503 errors with the message

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Screenshots

NOTE: The endpoint stopped responding around 15:30 in the graphs below.

Monitoring nr of bytes in:

Nr of requests:

Stack traces

Nothing useful, just:

2020-08-16 05:38:34.697979:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:37.643022:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:40.577522:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:42.008412:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:43.513294:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:45.425255:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:48.327276:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:38:51.316962:cortex:pid-447:INFO:200 OK POST /predict
2020-08-16 05:38:54.009212:cortex:pid-447:INFO:200 OK POST /predict
2020-08-16 05:38:55.852878:cortex:pid-447:INFO:200 OK POST /predict
2020-08-16 05:38:57.525264:cortex:pid-447:INFO:200 OK POST /predict
2020-08-16 05:39:00.795236:cortex:pid-447:INFO:200 OK POST /predict
2020-08-16 05:39:04.437013:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:05.981920:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:09.314293:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:12.343143:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:15.821708:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:19.083554:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:22.048843:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:24.943968:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:26.613330:cortex:pid-448:INFO:200 OK POST /predict
2020-08-16 05:39:29.702703:cortex:pid-448:INFO:200 OK POST /predict

Additional context

• The prediction takes about ~150ms on my Dell with Intel© Core™ i7-8750H CPU @ 2.20GHz × 6, 32GB Ram.
• All the load balancer target are marked as "unhealthy", even when they work (i.e. I can send requests and receive 2XX responses)
• The load balancer healthcheck endpoint returns the following

/healthz
{
        "service": {
                "namespace": "istio-system",
                "name": "ingressgateway-operator"
        },
        "localEndpoints": 0
}%  

Suggested solution

(optional)

Description

Add support for exporting environment variables from an .env file placed in the root directory of a Cortex project.

Motivation

In case the user doesn't want to export environment variables using the predictor:env field in cortex.yaml. A reason for that could be to keep the cortex.yaml deployment clean.

When deploying an API and observing logs, it seems that the most time-consuming part of deployment is:

2021-01-25 18:37:27.401057:cortex:pid-1:INFO:downloading the project code
2021-01-25 18:37:27.483562:cortex:pid-1:INFO:downloading the python serving image

Is there a way to somehow make deploying an API quicker?

We are trying to deploy a text generation API on AWS. We do not expect the API to receive a lot of traffic initially and hence we would like to save some costs. My idea was that min_replicas can be set to 0 which would not keep an instance idle in case the traffic on the API is none. As soon as a new request would come in cortex would spawn a new instance and shut it down once the traffic goes back to 0.

However, I noticed that setting min_replicas to 0 is invalid. Isn't the above use case a valid one for this? Also, is this a recent change? I vaguely(very) remember that this was possible to do in version 0.20(Please correct me if I'm wrong) but it seems like it is not in 0.26.

cc @deliahu I opened a new thread here because - 1) It's a different issue than the other thread , 2) Other users might benefit from the conversation here.

Changes

• Fix typo: async_queue_length -> async_queued so that the list of api_names is populated (currently empty)
• Use =~ with api_name where missing to enable displaying multiple AsyncAPIs on a panel
• For the "In-Flight Requests" panel include the api_name in the legend

Testing

I have made the corresponding updates manually through the Grafana UI for our deployed Cortex cluster. AsyncAPIs now list in the "Cortex / AsyncAPI" dashboard and the dashboard works when multiple AsyncAPIs are selected.

checklist:

• [ ] run make test and make lint
• [ ] test manually (i.e. build/push all images, restart operator, and re-deploy APIs)
• [ ] update examples
• [ ] update docs and add any new files to summary.md (view in gitbook after merging)
• [ ] cherry-pick into release branches if applicable
• [ ] alert the dev team if the dev environment changed

I don't really know how to word it correctly, long story short, I need to use the "http://$URL/" instead of http://$URL/$API_NAME" for one of the multiple APIs inside the cluster, I haven't found any way to do it in the documentation, but surely it is implemented.

Bumps sigs.k8s.io/aws-iam-authenticator from 0.5.3 to 0.5.9.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Description

Replace cloud provider specific code in cortex cluster commands by using the CDK API.

Motivation

Make cluster management commands more independent from each cloud provider. Make it easier to use code to define the infrastructure (aka Cortex) in this case.

Description

As it is described in https://docs.cortex.dev/clusters/management/auth#minimum-iam-policy, the current minimum IAM policy is to grant the cortex CLI (and by that extension to eskctl) full control over the EC2/EKS services.

Motivation

These should be restricted to a resource-based policy that would limit what an IAM role/user can do. This is especially helpful in bigger corporations where there are more than a handful of developers and the company's policy on what access its devs have is more stringent.

Additional context

This seems to be blocked on what eksctl requires: https://eksctl.io/usage/minimum-iam-policies/. Talk to the eksctl team to see if there's a way to further reduce the IAM policy requirements.