Looking for Jira Data Center to Cloud Migration Tips?

Preparing

There is few thing to do before the first test migration:

Disable mail
Limit access
Install Apps - Notice this is not like Data Center - You cant disable and enable them....

Clean up

I do recommend to clean up first:

Backup and delete spaces
Use Scriptrunner to purge old versions of pages; can also be done via the API.
Look for attachments, sometimes an On Premise Confluence has been used as a storage for files, so some pages may have a lot of (huge) attachments - and since there are no requirements in Confluence regading that attachments must be on a page or referenced, they just build up and people tend to see the page as a storage/drive.
Investigate User Macro's - those are not supported in cloud (see below). Delete all that can be deleted.

Nested Macros

My main problem was in overall Nested Macro's - in abundance... Like this (and via very complex User Macro code):

In the cloud nesting in also (mostly) possible in very few cases due to the security model of apps; - maybe Forge will improve this.

But Nested Macros are problematic, typically pages are not converted to the new editor, making them almost impossible to edit.

Notice this bug: https://jira.atlassian.com/browse/CONFCLOUD-80511 - That Atlassian "Wont Fix" - Its not just Expand in panels, but most Macro's in panels.

Replacing User Macros

The best tip is the "User Macro for Confluence Cloud" App; as this will help in 3 parts:

Actually migrate User Macros
Make Macro replacements for un-migratable macros or Apps that You dont want in Cloud
Make Dummy Macros to mitigate non-existing or non-functioning macros.

One of the great parts of the App is tha You can define the Macro name Youself - opposite Scriptrunner, where the Macro is always named sr-macro-some-uuid

Nesting is still not supported here.

Scriptrunner Macros

Quite a lot of scriptrunner Macros can be converted quite a lot without changes, the XHTML output sould not be changed, for me it was mostly to change API and external calls to Unirest.

Unfortunately, Scriptrunner Macro names cant be set in cloud, its an automatic "sr-uuid" naming convention, why You might have to change the souce content in the Data Center- See below under Manipulations in the Database

Macro removal via Scriptrunner

To remove a macro in one or more pages, use this script (TAKE BACKUP FIRST).

Corrections according to https://community.atlassian.com/forums/Confluence-questions/Macro-removal-in-many-pages/qaq-p/2981874?utm_source=atlcomm&utm_medium=email&utm_campaign=mentions_reply&utm_content=topic#U3024976

import com.atlassian.confluence.pages.PageManager
import com.atlassian.sal.api.component.ComponentLocator
import org.jsoup.Jsoup

def space = '...'
def pageName = '...'

def pageManager = ComponentLocator.getComponent(PageManager)
def page = pageManager.getPage(space, pageName)

log.warn "Inspecting page ${page.title}"

def body = page.bodyContent.body
def parsedBody = Jsoup.parse(body)

// Remove the unwanted macros, we wanted to remove these three
parsedBody.select('ac|structured-macro[ac:name=getlastmodifiername]').remove()
parsedBody.select('ac|structured-macro[ac:name=getlastdatemodified]').remove()
parsedBody.select('ac|structured-macro[ac:name=getpageversion]').remove()

// Set prettyPrint to false to not break macro configurations with spaces and line breaks
parsedBody.outputSettings().prettyPrint(false)

// log.warn parsedBody.toString()
// Save the page

pageManager.saveNewVersion(page) { pageObject ->
    pageObject.setBodyAsString(parsedBody.toString())
}

Manipulations in the database

To change for exaple macro names, run SQLs again the database before migration (TAKE BACKUP FIRST). Here I do replace some User Macro names with names of a new Scriptrunner Macro in Cloud.

\c confluence;
update bodycontent set body=REPLACE(body,'ac:name="jiracsdb"','ac:name="sr-macro-97d59fcd-ce9a-4df3-a4db-c2deb57e36e6"') where body LIKE '%"jiracsdb"%';

Yes, I am aware this is ALL content, not just the latest version. Remember to restart and reindex afterwards.

Add pages in Data Center

This is based on a selection og Home pages via a category "customer" in the Space Directory and a selection in the Confluence Database

All Home pages are added 2 pages with the Storage Format code from 2 other pages (Id=480590905 and 480590813) (TAKE BACKUP FIRST):

addPages.sh

#!/bin/bash
# set -x

# Query strings
SQL="SELECT s.homepage || '#' || s.SPACEKEY FROM CONTENT_LABEL cl LEFT JOIN LABEL l ON (l.LABELID = cl.LABELID) LEFT JOIN CONTENT c ON (c.CONTENTID = cl.CONTENTID) LEFT JOIN SPACES s ON (s.SPACEID = c.SPACEID) WHERE l.NAMESPACE = 'team' AND l.NAME = 'customer';"

# Declare and instantiate the space* arrays
declare -a spaceHomepage="`echo $SQL | psql -t -h localhost -U confluenceuser -d confluence`"

# Iterate through each Space
for sID in ${spaceHomepage[@]}
do
   #Split $sID

   HOMEPAGEID=$(echo $sID | cut -d "#" -f 1)
   SPACEKEY=$(echo $sID | cut -d "#" -f 2)

   # Customer Details
   rm page1.json
   curl -u username:password -X GET https://server.domain.com/rest/api/content/480590905?expand=body.storage > page1.json

   rm customer-details.xml
   cat page1.json | jq .body.storage.value > customer-details.xml
   CUSTOMERDETAILS=$(cat customer-details.xml)

   # Customer Information
   rm page2.json
   curl -u username:password -X GET https://server.domain.com/rest/api/content/480590813?expand=body.storage > page2.json

   rm customer-information.xml
   cat page2.json | jq .body.storage.value > customer-information.xml
   CUSTOMERINFORMATION=$(cat customer-information.xml)

   #Add Customer Information page
   echo "Adding Customer Information page for space ${SPACEKEY} - ${HOMEPAGEID}"
   echo "{\"type\":\"page\",\"space\":{\"key\":\"${SPACEKEY}\"},\"ancestors\":[{\"id\":${HOMEPAGEID}}],\"body\":{\"storage\":{\"value\":${CUSTOMERINFORMATION},\"representation\":\"storage\"}},\"title\":\"Customer Information\"}" > data.json


   curl -u username:password -X POST -H 'Content-Type: application/json' --data @data.json  https://server.domain.com/rest/api/content/

   #Add Customer Details page
   echo "Adding Customer Details page for space ${SPACEKEY} - ${HOMEPAGEID}"
   echo "{\"type\":\"page\",\"space\":{\"key\":\"${SPACEKEY}\"},\"ancestors\":[{\"id\":${HOMEPAGEID}}],\"body\":{\"storage\":{\"value\":${CUSTOMERDETAILS},\"representation\":\"storage\"}},\"title\":\"Customer Details\"}" > data.json

   curl -u username:password -X POST -H 'Content-Type: application/json' --data @data.json  https://server.domain.com/rest/api/content/

done

Confluence willl reindex automatically.

Change Space Homes before Migration

This is based on a selection of Home pages via a category "customer" in the Space Directory and a selection in the Confluence Database

Each Home pages are updated with the Storage Format code from another page (Id=480590449) (TAKE BACKUP FIRST):

changeHome.sh

#!/bin/bash
# set -x

# Query strings
SQL="SELECT s.homepage || '#' || s.SPACEKEY FROM CONTENT_LABEL cl LEFT JOIN LABEL l ON (l.LABELID = cl.LABELID) LEFT JOIN CONTENT c ON (c.CONTENTID = cl.CONTENTID) LEFT JOIN SPACES s ON (s.SPACEID = c.SPACEID) WHERE l.NAMESPACE = 'team' AND l.NAME = 'customer';"

# Declare and instantiate the space* arrays
declare -a spaceHomepage="`echo $SQL | psql -t -h localhost -U confluenceuser -d confluence`"

# Iterate through each Space
for sID in ${spaceHomepage[@]}
do
   #Split $sID

   HOMEPAGEID=$(echo $sID | cut -d "#" -f 1)
   SPACEKEY=$(echo $sID | cut -d "#" -f 2)

   # Page Id of the page we want copy copy Storage Format from
   FRONTSOUCE="480590449"

   # Get newFront
   rm page1.json
   curl -u username:password -X GET https://server.domain.com/rest/api/content/${FRONTSOUCE}?expand=body.storage > page1.json

   cat page1.json | jq .body.storage.value > front.xml
   FRONT=$(cat front.xml)

   # Get existing version
   rm page2.json
   curl -u username:password -X GET https://server.domain.com/rest/api/content/${HOMEPAGEID}?expand=version > page2.json

   TITLE=$(cat page2.json | jq .title)
   VERSION=$(cat page2.json | jq .version.number)
   VERSION=$(($VERSION+1))

   #Replace front page
   echo "Replace front for space ${SPACEKEY} - ${HOMEPAGEID}"
   echo "{\"id\":\"${HOMEPAGEID}\",\"type\":\"page\",\"space\":{\"key\":\"${SPACEKEY}\"},\"version\":{\"number\":$VERSION},\"body\":{\"storage\":{\"value\":${FRONT},\"representation\":\"storage\"}},\"title\":$TITLE}}" > data.json

   curl -u username:password -X PUT -H 'Content-Type: application/json' --data @data.json  https://server.domain.com/rest/api/content/$HOMEPAGEID

done

Confluence willl reindex automatically.

Copying a LOT of fields

This bash script can copy a lot of fields from one "special field" to a text field, looping:

All projects
All Issuetypes for a project
An Array of fields

I use it for +1 million values - across +200 projects, 60 Issuetypes and +20 fields (from the Elements Connect App)

This is used for https://doc.elements-apps.com/elements-connect-cloud/04-how-to-copy-migrated-values-into-connected-fiel -as the current Migration provided by the App Vendor really sucks bigtime....

Secondly, any groovy running within Jira will timeout (eventually) and typically we are unsure if the jobs continues behin the scenes etc.

The FIELDMAP= is an array of sourceid|destinationid|type - where the latter is S=Single Text, M=Multivalue

The "Automation" user must have at least "Browse Project" and "Edit Issue" permissions

Do run the script in a screen/tmux to avoid it being stopped by connection issues.

config.txt

USERNAME=automation
PASSWORD=*******
BASEURL="https://server.domain.dk"

UpdateShadowFieldsForIssue.sh

#!/bin/bash

mkdir runlog

source config.txt

IFS=$(echo -en ",")

REST="/rest/scriptrunner/latest/custom/UpdateShadowFieldsForIssue"
DRYRUN=""
EMPTYONLY="1"
SKIPINDEX=""
DAYSBACK=

#Loop projects
curl -s -u "$USERNAME:$PASSWORD" -X GET -H 'Content-Type: application/json'  $BASEURL/rest/api/2/project?maxResults=1000 > projects.json
cat projects.json | jq '.[].key' | while read -r PROJECTKEY; 
do

  PROJECTKEY=$(echo $PROJECTKEY | sed 's/\"//g')
  echo "$PROJECTKEY" >> runlog.txt

  #Loop Issuetypes for the Project
  curl -s -u "$USERNAME:$PASSWORD" -X GET -H 'Content-Type: application/json'  $BASEURL/rest/api/2/issue/createmeta/$PROJECTKEY/issuetypes?maxResults=1000 > issuetypes.json
  cat issuetypes.json | jq '.values[].name' | while read -r ISSUETYPE;
  do

    ISSUETYPE=$(echo $ISSUETYPE | sed 's/\"//g')
    echo $ISSUETYPE >> runlog.txt

    #Loop Fields
    declare -a FIELDMAP=("23321|29033|M" "23523|29020|S" "24523|29021|S" "23522|29022|S" "24623|29026|S" "24422|29023|S" "23726|29024|S" "23728|29034|M" "23729|29035|M" "23727|29025|S" "24920|29036|S" "24921|29037|S" "24624|29027|S" "24423|29028|S" "23320|29032|S" "24821|29031|S" "24625|29030|S" "24424|29029|S" "24922|29039|S" "24923|29038|S" "24924|29040|S" "24925|29041|S" "24926|29042|S" "25028|29043|S" "25027|29044|S")

    for FIELD in "${FIELDMAP[@]}"
    do
	
      #echo "Field $FIELD"
	    SOURCEFIELDID=$(echo $FIELD | cut -d '|' -f 1)
	    SHADOWFIELDID=$(echo $FIELD | cut -d '|' -f 2)
	    TYPE=$(echo $FIELD | cut -d '|' -f 3)

      ISSUETYPE=$(echo $ISSUETYPE | sed 's/ /%20/g')
	
	    QUERYSTRING="projectkey=$PROJECTKEY&issuetype=$ISSUETYPE&sourcefieldid=$SOURCEFIELDID&shadowfieldid=$SHADOWFIELDID&type=$TYPE&emptyonly=$EMPTYONLY&dryrun=$DRYRUN&skipindex=$SKIPINDEX&daysback=$DAYSBACK"
	    echo "curling.... $QUERYSTRING"
	    echo "Running.... $QUERYSTRING" >> log.txt
	    curl -u "$USERNAME:$PASSWORD" -o runlog/$PROJECTKEY-$ISSUETYPE-$SOURCEFIELDID -X GET -H 'Content-Type: application/json' "$BASEURL$REST?$QUERYSTRING"

    done
  done
done

And the Scriptrunner REST service needed:

import com.atlassian.jira.component.ComponentAccessor
import com.atlassian.jira.issue.Issue
import com.atlassian.jira.issue.IssueManager
import com.atlassian.jira.user.util.UserUtil
import com.atlassian.jira.issue.CustomFieldManager
import com.atlassian.jira.issue.fields.CustomField
import com.atlassian.jira.bc.issue.search.SearchService
import com.atlassian.jira.issue.search.SearchProvider
import com.atlassian.jira.issue.search.SearchResults
import com.atlassian.jira.web.bean.PagerFilter
import com.atlassian.jira.user.ApplicationUser
import com.atlassian.jira.event.type.EventDispatchOption
import groovy.json.JsonSlurper
import com.atlassian.jira.issue.index.IssueIndexingService
import com.atlassian.jira.issue.MutableIssue
import com.atlassian.jira.issue.IssueManager
import com.atlassian.jira.util.ImportUtils
import groovy.transform.BaseScript
import com.onresolve.scriptrunner.runner.rest.common.CustomEndpointDelegate
import javax.ws.rs.core.MultivaluedMap
import javax.servlet.http.HttpServletRequest
import javax.ws.rs.core.Response

@BaseScript CustomEndpointDelegate delegate

UpdateShadowFieldsForIssue(httpMethod: "GET", groups: ["automaticservices","jira-administrators"]) { MultivaluedMap queryParams ->

    Random random = new Random()
    String scriptRunIdent = Math.abs(random.nextInt() % 99999) + 1
    String scriptName = "UpdateShadowFieldsForIssue.groovy"

    String projectKey = queryParams.getFirst("projectkey") as String
    String issueType = queryParams.getFirst("issuetype") as String
    String sourceFieldId = queryParams.getFirst("sourcefieldid") as String
    String shadowFieldId = queryParams.getFirst("shadowfieldid") as String  
    String type = queryParams.getFirst("type") as String
    String emptyOnly = queryParams.getFirst("emptyonly") as String
    String dryRun = queryParams.getFirst("dryrun") as String
    String skipIndex = queryParams.getFirst("skipindex") as String
    String daysBack = queryParams.getFirst("daysback") as String
    Integer currentNumber = 0
    def builder = new groovy.json.JsonBuilder()

    String jql="Project=" + projectKey + " and issuetype='" + issueType + "' and cf[" + sourceFieldId + "] is not empty"
    if (emptyOnly == "1")
    {
        jql=jql + " and cf[" + shadowFieldId + "] is empty"
    }
    if (daysBack != "")
    {
        jql=jql + " and updated > -" + daysBack + "d"
    }
    log.info "Script=" + scriptName + " ScriptRunIdent=" + scriptRunIdent + " Message='JQL: " + jql + "'"
    
    ApplicationUser currentUser = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser()
    CustomFieldManager customFieldManager = ComponentAccessor.getCustomFieldManager()
    IssueManager issueManager = ComponentAccessor.getIssueManager()
    def issueIndexingService = ComponentAccessor.getComponent(IssueIndexingService)

    CustomField sourceField = customFieldManager.getCustomFieldObject("customfield_" + sourceFieldId)
    String jiraSourceFieldType =sourceField.getCustomFieldType().getName()
    CustomField shadowField = customFieldManager.getCustomFieldObject("customfield_" + shadowFieldId)
    String jiraShadowFieldType = shadowField.getCustomFieldType().getName()

    log.info "Script=" + scriptName + " ScriptRunIdent=" + scriptRunIdent + " Message='Source field Jira type: " + jiraSourceFieldType + "'"
    log.info "Script=" + scriptName + " ScriptRunIdent=" + scriptRunIdent + " Message='Shadow field Jira type: " + jiraShadowFieldType + "'"

    if (jiraShadowFieldType == "Text Field (single line)" || jiraShadowFieldType == "Text Field (multi-line)")
    {
        SearchService searchService = ComponentAccessor.getComponent(SearchService.class)
        SearchService.ParseResult parseResult =  searchService.parseQuery(currentUser, jql)

        if (parseResult.isValid())
        {
            SearchResults results = searchService.search(currentUser, parseResult.getQuery(), PagerFilter.getUnlimitedFilter())
            final List issues = results?.results
            totalIssues = issues.size()

            issues.each { theIssue ->

                log.info "Script=" + scriptName + " ScriptRunIdent=" + scriptRunIdent + " Message='Value " + theIssue.getCustomFieldValue(sourceField) + "'"

                //Extract value(s) here from Source field. Its possible to expand for another 'jiraSourceFieldType'
                if (jiraSourceFieldType.contains("Elements Connect"))
                {
                    sourceFieldValue = extractElementsValues(theIssue.getCustomFieldValue(sourceField),type)
                }
                else
                {
                    sourceFieldValue = theIssue.getCustomFieldValue(sourceField)
                }
                shadowFieldsValue = theIssue.getCustomFieldValue(shadowField)

                if (sourceFieldValue != shadowFieldsValue)
                {
                    currentNumber = currentNumber + 1
                    
                    if (dryRun != "1")
                    {
                        log.info "Script=" + scriptName + " IssueKey=" + theIssue.getKey() + " ScriptRunIdent=" + scriptRunIdent + " Message='Copying " + sourceFieldId + " to " + shadowFieldId + ", value: " + sourceFieldValue + "'"
                        MutableIssue mIssue = issueManager.getIssueByCurrentKey(theIssue.getKey())   
                        try {
                            mIssue.setCustomFieldValue(shadowField,sourceFieldValue)
                            ComponentAccessor.getIssueManager().updateIssue(currentUser, mIssue, EventDispatchOption.DO_NOT_DISPATCH, false)
                        }
                        catch  (Exception ex)
                        {
                            log.info "Script=" + scriptName + " IssueKey=" + theIssue.getKey() + " ScriptRunIdent=" + scriptRunIdent + " Error='Error Updating: " + ex.getMessage() + "'" 
                        }
                        //Reindex Issue
                        if (skipIndex != "1")
                        {
                            boolean wasIndexing = ImportUtils.isIndexIssues()
                            ImportUtils.setIndexIssues(true)
                            issueIndexingService.reIndex(mIssue)
                            ImportUtils.setIndexIssues(wasIndexing)
                        }
                    }
                    else
                    {
                        log.info "Script=" + scriptName + " IssueKey=" + theIssue.getKey() + " ScriptRunIdent=" + scriptRunIdent + " Message='DryRun - Copying " + sourceFieldId + " to " + shadowFieldId + ", value: " + sourceFieldValue + "'"
                    }
                }
                else
                {
                    log.info "Script=" + scriptName + " IssueKey=" + theIssue.getKey() + " ScriptRunIdent=" + scriptRunIdent + " Message='Skipped copy due to identical values'"
                }    
            }
            if (dryRun != "1")
            {
                if (currentNumber > 0)
                {
                    builder  {success "Fields copied successfully. Copied " + currentNumber + " field(s)."}
                }
                else
                {
                    builder  {success "No Fields copied."}
                }            
            }
            else
            {
                builder  {success "Dry Run. Woulf have copied " + currentNumber + " field(s)."}
            }
            Response.status(200).entity(builder.toString()).build()
        }
        else
        {
          builder  {error "Not valid JQL"}
          Response.status(500).entity(builder.toString()).build()
        }
    }
    else
    {
      builder  {error "Shadow field is not a Text Field."}
      Response.status(500).entity(builder.toString()).build()
    }
}

def extractElementsValues(theValue,fieldType)
{
    if (theValue == null)
    {
        return null
    }

    def slurper = new JsonSlurper().parseText(theValue)
    if (fieldType == "M")
    {
        slurper.keys.join("\n").toString()
    }
    else
    {
        slurper.keys[0].toString()
    }   
}

You can test the script with:

curl -u "automation:********" -X GET -H 'Content-Type: application/json' 'https://server.domain.dk/rest/scriptrunner/latest/custom/UpdateShadowFieldsForIssue?projectkey=SUPPOORT&issuetype=Task&sourcefieldid=23522&shadowfieldid=29022&type=S&emptyonly=1&dryrun=&skipindex=&daysback='

In the "runlog" directory below the script location, there with be a file for each PROJECT-ISSUETYPE-SOURCEFIELDID - and the content will be the REST output

Testing

You can test migrate spaces to Cloud as many times as needed.

There is another bug, where deletion of spaces fails see https://jira.atlassian.com/browse/CONFCLOUD-80839. Only solution is to make a ticket via Atlassian Support..

Comala Document Management Migration

The state of the Cloud App is very feature less compared with the Data Center App.

Be aware that the Cloud App has a huge difference between "Space Workflows" and "Page Workflow" - even though the same workflow is applica to a page:

Applied Automatically via active workflow or labels - Workflow Parameters cant be changed and the backend parameters are fixed to the page.
Applied manual via the "Add workflow" funtion in the byline, parameters can be changed.

Content

Space Tools

Confluence Data Center to Cloud Migration Tips