Currently Optimus Supports sensors for job dependencies which are within and the outside the project but they are managed by the same Optimus Server. It would be helpful if Optimus supports job sensors which are managed in a different Optimus, as with in an organisation there will be many deployments, checking for data availability may not always guarantee completeness & correctness of data which is guaranteed through Optimus dependencies.

Expectation : The sensor provides checks for the status of the jobs b/w the input window.

Configuration :

dependencies : 
 job : 
 type : external
 project : 
 host : 
 start_time : // start time of the data that the job depends on
 end_time : // end time of the data that the job depends on.

The Optimus Server which accepts the requests based on its window, schedule configuration checks for all the jobs which outputs the data for the given window

This has the challenge of breaking the dependencies when job name changes.

The current cmd package contains many implementations in the scope of accepting the user input on how to execute Optimus. There are some issues being observed for the current approach:

• many functionalities are defined within one package, making the package itself seems bigger with a lot of functions, variables, and constants that are accessible accross the package
• with many components defined within one package, some of it could conflict with one another, and there were cases during development that two components (in this example, variables) were defined but served the same purpose
• during development, IDE or text editor's suggestion could clutter with other functionalities

To address the mentioned issues above, one approach that can be done is by restructuring the package to grouping some similar functionalities, like for job command, it is put in one place like struct and/or package.

Currently, project & namespace cannot be register through Optimus CLI, as most of the users are CLI users, it would be better if there is a mechanism to register project & namespace.

Acceptance Criteria

1. User should be able to register a project without any namespace.
2. User should be able to register a namespace only.
3. User should be able to register both project & namespace together.
4. On deploy if any new project/namespace is modified it should be register/updated.
5. remove existing config init command

User experience

1. optimus project/namespace register

Hi Maintainers,

I need to organizing my dags/jobs on the Airflow UI, there's already that feature in airflow using tags https://airflow.apache.org/docs/apache-airflow/stable/howto/add-dag-tags.html and there's already labels on job spec

so, I'll need that labels rendered as tags in optimus' rendered dag codes. I hope my code can be tested and reviewed to give this feature on optimus.

thank you

As part of this card, we would expect observers usage and implementation to be standardized, currently the way observers is used involves some processing on the client side, rather it would be better if through observers all the information is passed in a direct consumable fashion such that clients just log.

Scope

1. Events are not standardized - event naming.
2. Optimus CLI we are extra processing after consuming these events rather we can avoid and just the log the events.

Describe the solution you'd like An Optimus sensor should check given a resource and the start and end dates, will check the corresponding jobs are successful or not. If Optimus Sensor is used in the Optimus setup then it should be automatically inferred.

Users

1. Other Optimus Users with in the same organization managing a different
2. Users using a different system other than optimus for managing the pipelines

Describe alternatives you've considered Relying on respective storage sensors with automated inference, but it has its own challenges around data completeness and data quality.

• [x] #398
• [x] #399
• [x] #400

• [x] overriding the config via flags provided by cobra
• [x] mapping the flag with - delimiter instead of . delimiter (--project-name instead of --project.name)
• [x] change flag names for each command that needs config overriding

TODO next (can be done in separate PR):

• [ ] bump salt version to support pflags
• [ ] rename --project to --project-name on entrypoint.sh for each plugins

Users should be able to do backup and replay for downstream jobs with a different namespace, as long as authorized to do so.

• Optimus CLI will accept allowed downstream namespaces that will be replayed/backup (as a flag)
• "*" means downstream from all namespaces (within the same project) are allowed
• the default will be empty. means, the allowed downstream are only from same namespace

Also, adding ignore downstream option in Replay.

Description Core scheduler management is not properly encapsulated and can be accessed and modified everywhere. We need to encapsulate the scheduling functions to only be accessible by proper interfaces. As part of this we will introduce a domain for dealing with core scheduling bounded context.

Depends on : 575

Acceptance Criteria

• [x] core scheduling related logic should be encapsulated
• [x] Client should still rely on the same apis & there should be minimal change in the api contract.
• [x] All service and handlers related to job runs will be part of this domain
• [x] Remove the old api from Protos.
• [x] Dag compilation should happen properly.
• [x] Dag uploading/updating and deletion of Dags for removed jobs should happen properly.
• [x] Priority resolution logic should not be changed.
• [x] DAG deploying to Scheduler should be run independently of job spec creation, and dependency and priority resolution
• [x] Alerting should be proper for SLA misses and Job Failures(only complete DAG failures/not operator failures)
• [x] Relevant metrics and logging should be improved to address the debuggability concerns.
• [x] Re assess the metric collection callbacks/events from the scheduler.
• [x] Re assess the Data Base model for metrics collected for job_run and its breakup(sensors/tasks/hooks).

Scope

• [x] Introduction of the new scheduler bounded context with all the handlers & services.
• [x] JobRunInput
• [x] UploadToScheduler
• [x] RegisterJobEvent [takes care of job state updation( scheduling metrics ) and alerting ]
• [x] JobRun
• [x] Introduction of new repositories to work with the new models being introduced as part of this BC.
• [x] Improve the debuggability through logging, capturing the right metrics.

Out of Scope

1. Handling of replay functionality in Optimus
2. Handling of backup
3. GetJobTask API
4. GetWindow API
5. JobStatus API
6. RunJob API

Describe the bug For any job, if we configure the slack alerts on failure, then for event type such as task, sensor, hook failure the slack alerts are not getting posted successfully.

To Reproduce Steps to reproduce the behavior:

1. Configure a job behaviour to notify on slack channel on failure
2. Run the job and while the job is running, mark any task as failed.

Expected behavior Failure message on configured slack channel.

Started happening from tag v0.3.0

Background

If we look at the latest commit (referring to this), internally we will have migration mechanism being used. The migration up is executed whenever we run the server. However, if we check even further, no down migration is provided. So, even if there's an issue with the current database schema, it's quite tricky to roll back.

Proposal

To address this, this issue proposes to have such functionality being provided in the Optimus. At the high level, more or less, it will be like the following:

optimus server migrate up
# it will execute all migrate `up`

optimus server migrate down
# it will execute all migrate `down`

optimus server migrate {n}
# n is integer number, with positive means `up` n-time and negative means `down` n-time

The command or the mechanism is flexible, but the point is that the up and down are both defined properly.

Additional Context

Since the mentioned commit use golang-migrate/migrate, we can use .Steps(int) method for the n up or down times.

This PR is to update the benchmark tests based on the latest implementation. The benchmark tests being updated:

• repositories for tenant (like secret, project, and namespace)
• repositories for resource (like resource and backup)
• repositories for job
• repositories for scheduler (like job run)

Is your feature request related to a problem? Please describe. Optimus currently lacks ability to dump all the jobs present in a project, it can be used for inspecting or backing up the specs present in database. This can also include jobs which are deleted, can be sent back as archived.

Describe the solution you'd like Add command to download all the resources and jobs in a project as yaml files.

Describe alternatives you've considered Currently no other alternative other than taking a dump of database.

Describe the bug GetJobSpecification and GetJobSpecifications are not returning 404 if the job is not found.

Expected behavior

• If no job is found, should return 404 instead of 200 with an empty job.
• Any caller can handle it gracefully based on the need
• For external resource managers (being used when resolving job upstreams externally), if receiving a 404 response it will not return an error.

Describe the bug We are comparing the current optimus version and the previous version to decide whether to run migration or not. But the version is string so, the comparison behaviour is not guaranteed. eg 0.4.0-rc1 is greater than 0.4.0

Expected behavior Migrations should run on the deployment of new release.

Is your feature request related to a problem? Please describe. Optimus currently runs with version 2.1.4 of airflow, it should support the newer versions of airflow.

Describe the solution you'd like Currently the pod launcher used in base_dag.py in optimus uses pod_launcher which is deprecated in newer versions of airflow, hence the same dag will not work with never versions. We need to update the SuperKubernetesOperator to provide support for newer version of airflow.

Description The plugin command, optimus plugin install, which will be used on the server side should belongs to the server package. As part of making the client and server segregation, this command should be moved to the server command package

Acceptance Criteria

• [ ] Command optimus plugin install should be moved to the server command package

Out of Scope N/A

Tech Details TBD