V5 tasks and other enhancements

[maxceem]

V5 tasks

1. [Easy] Create a script to Index Timelines to ES, validate that code works well with data from ES and deploy.

2. [Easy but blocked] Remove support for bypassing ES when getting Timelines by id. This could be done after avoiding making cascading updates in milestones Avoid cascading updates for milestones server-side and move logic to client appirio-tech/connect-app#3709.

3. [A lot here] Support having new models in ES by Support having new models in ES #432

V5 enhancements

4. [Need some good planing] Get rid of RabbitMQ in favor of using Kafka or some other "events".

5. Create a script for comparing data in DB/ES and alert us. - in the process now.

6. [Need planning, better to implement after getting rid of RabbigMQ] Unify sending events to Kafka:

   • we should agree on format of the message, for example we may want to send original and updated objects during updates, not only updated like now
   • in some places currently, we send only "updated" fields during updates, in some full objects. Also, during deleting sometimes we send full objects, sometimes only "id". Again, we should agree on the format and follow it in all endpoints.
   • currently, we have a several methods to send Kafka events to reindex data, and they quite tricky go through several methods/events, before actually being sent. We have to create a single simple method that would directly send to Kafka to keep things clear. It should fit all our cases. Most likely we can do that after we get rid of RabbitMQ, as so far we have our complex way as some events have to be also processed by RabbitMQ.

7. [Looks straightforward, can start] Improve unit tests for GET endpoints so they test both cases: getting data from ES and gettings data from DB. So far most unit tests only tests getting data from DB, while some requests could be quite complex, and have filtering and so on. If we have methods to reindex any data fast, we can write unit tests a universal way so they could be applied to both DB and ES. See this unit test implemented for POC: https://github.com/topcoder-platform/tc-project-service/blob/develop/src/routes/orgConfig/list.spec.js#L134-L153. It works good for this endpoint, and we can try to apply the same approach for all other GET endpoints.

8. For the PUT/POST endpoints we may consider to always test that correct messages are sent to Kafka. Though it feels that efforts for this wouldn't pay off and we only have to test complex cases where multiple Kafka messages might be sent https://github.com/topcoder-platform/tc-project-service/blob/develop/src/routes/milestones/update.spec.js#L1284-L1473 which we already do.

9. Should we update tc-notifications to use general Kafka topics instead of our custom topics somehow?

10. Invite related endpoints should follow RESTful format, now they don't and they have very custom logic. This would require updating the logic server and client-side.

Other enhancements

• Improve demo data seed script. Right now when we run it creates a lot of data using Project Service API endpoints, data is created immediately in DB, but it takes 5 - 10 minutes to get processed by project-processor-es.
  Also, I'm still in favor or keeping demo data locally, rather download it from DEV env each time, so we don't have to provide CONNECT_APP_USER_TOKEN, which is confusing to members sometimes and not comfortable in practice, as this command has to be run multiple times and every time we have to provide a fresh token.

Permissions

During testing of V5 changes, it looked that it's hard to know for QA team where and which permissions users should have. Also, as there are more roles and different situations it's not always to get from the code what is the actual implementation from the user permission perspective.
We already have some tasks listed: #346 and in appirio-tech/connect-app#2841.
I guess we should continue on them and also add one requirement:

• our configs of permission should be easily read by humans, so the QA team and us, could use our config files with permission to easily understand what permissions are actually implemented for each case. Also, for the QA team, we may think about generating from configs nice-looking HTML files especially if config files not always present information in a handy way.