Master Data Upload process

When a Master Data Upload file (Excel or JSON) is uploaded, the data are processed in the following order:

1. Create sites
2. Update sites
3. Create sources
4. Update sources
5. Move sources to site
6. Create events
7. Update events
8. Create users
9. Update users

Warning

By design, Opinum storage processes files uploaded to the storage in parallel. This means that if multiple Master Data Upload files are uploaded in the storage at "the same time", they will be handled in parallel which could cause unpredictable behavior, e.g. creating multiple entities with the same business key.

It is strongly recommended not to upload multiple Master Data Upload files at the same time.

Create sites

Any site, identified by a business key, that does not exist is created

FOR each businessKey IN SiteSheet[column[_opSite.Key]]
  existingSite = DATAHUB_GET_SITE(businessKey)
  IF(existingSite IS NULL)
    siteDefinition = READ columns[_op*]
    CALL BeforeSiteCreation_Customization
    DATAHUB_CREATE_SITE(siteDefinition)
    CALL AfterSiteCreation_Customization
  ENDIF
ENDFOR

Update sites

Any site, identified by a business key, that exist and was not previously created is updated if it differs from the existing one

	FOR each businessKey IN SiteSheet[column[_opSite.Key]]
		existingSite = DATAHUB_GET_SITE(businessKey)
		IF(existingSite IS NULL)
			CALL “The site does not exist”
		ELSE
			siteDefinition = READ SiteSheet[columns[_op*]]
			IF(existingSite <> siteDefinition)
				CALL BeforeSiteUpdate_Customization
				DATAHUB_UPDATE_SITE(siteDefinition)
				CALL AfterSiteUpdate_Customization
			ENDIF
		ENDIF
	ENDFOR

Create sources

Any source, identified by a business key, that does not exist is created

	FOR each businessKey IN SourceSheet[column[_opSource.Key]]
		existingSource = DATAHUB_GET_SOURCE(businessKey)
		IF(existingSource IS NULL)
			sourceDefinition = READ SourceSheet[columns[_op*]]
			CALL BeforeSourceCreation_Customization
			DATAHUB_CREATE_SOURCE(sourceDefinition)
			CALL AfterSourceCreation_Customization
		ENDIF
	ENDFOR

Update sources

Any source, identified by a business key, that exist and was not previously created is updated if it differs from the existing one

	FOR each businessKey IN SourceSheet[column[_opSource.Key]]
		existingSource = DATAHUB_GET_SOURCE(businessKey)
		IF(existingSource IS NULL)
			CALL Error “The source does not exist”
		ELSE
			sourceSite = READ SourceSheet[columns[_opSite.Key]]
			IF(sourceSite IS NULL)
				CALL Error “The source’s site does not exist”
	ELSE
				sourceDefinition = READ SourceSheet[columns[_op*]]
				IF(existingSource <> sourceDefinition)
					CALL BeforeSourceUpdate_Customization
					DATAHUB_UPDATE_SITE(sourceDefinition)
					CALL AfterSourceUpdate_Customization
				ENDIF
			ENDIF
		ENDIF
	ENDFOR

Move sources to site

Any source, identified by a business key, that exist and is not attached to the site specified in the Master Data is moved to this Master Data site

	FOR each businessKey IN SourceSheet[column[_opSource.Key]]
		existingSource = DATAHUB_GET_SOURCE(businessKey)
		IF(existingSource IS NULL)
			CALL Error “The source does not exist”
		ELSE
			sourceSiteId = READ SourceSheet[columns[_opSite.Key]]
			IF(sourceSiteId IS NULL)
				CALL Error “The source’s site is invalid”
	ELSE
				IF(existingSource.SiteId <> sourceSiteId)
					existingSource.SiteId = sourceSiteId
					CALL BeforeSourceMove_Customization
					DATAHUB_UPDATE_SITE(existingSource)
					CALL AfterSourceMove_Customization
				ENDIF
			ENDIF
		ENDIF
	ENDFOR

Create users

Any user, identified by a business key, that does not exist is created

	FOR each businessKey IN UserSheet[column[_opSource.Key]]
		existingUser = DATAHUB_GET_USER(businessKey)
		IF(existingUser IS NULL)
			userDefinition= READ SourceSheet[columns[_op*]]
			DATAHUB_CREATE_USER(userDefinition)
		ENDIF
	ENDFOR

Update users

Any user, identified by a business key, that exist and was not previously created is updated if it differs from the existing one

	FOR each businessKey IN UserSheet[column[_opSource.Key]]
		existingUser = DATAHUB_GET_USER(businessKey)
		IF(existingUser IS NULL)
			CALL Error “The user does not exist”
		ELSE
			userDefinition= READ UserSheet[columns[_op*]]
			IF(existingUser <> userDefinition)
				DATAHUB_UPDATE_USER(userDefinition)
			ENDIF
		ENDIF
	ENDFOR