Copy Services Error

dickinsonzach

How do I go about reporting a bug?  We upgraded to 2.40 and received many of these:

STOR2RRD Copy Services Error for device gcdelta7kmy

name:source_vdisk_name:target_vdisk_name:group_name:status:progress:start_time

fcmap0:Genisys_iASP1_1:vdisk0::stopped:0:

The devices/relationship called out are the background Flash Copies when you have a Remote Copy setup between IBM V7000.  They go into a stopped state when not replicating.

So while true, it should be called out as an error.

Thank you, Zach.

Pavel

Hi,

do you say that stopped status is ok in this case?

dickinsonzach

It CAN be OK.  If you create a Global Mirror with Change Volumes.  It will create a Thin Volume where the changes are placed.  Then every 5 minutes (default), the changes are replicated from the change volume.  If there are no changes being made or very little, the relationship between the replicated volume and the Change Volume can be stopped.

On the back end, the relationship is a Flash Copy.  Watching it in real time the Flash Copies will go from active, to stopped, to active, etc...

Perhaps there needs to be an exception for Flash Copies that exist because they are part of a Remote Copy?

Pavel

is that related only to Global Mirror, right? Then stopped does nt meanerror status. That is clear. Metro mirror should not stop, right?

What about flash copy generally? That is your case, righ? Shall we behave same? i.e. stopped == ok

dickinsonzach

I cannot say for Metro Mirror.  The Global Mirror should not be stopped.



What can be stopped is the Change Copy Volume in the FlashCopy Mapping.


Does that make sense?  Thank you, Zach.

dickinsonzach

If there are any logs I can provide, please let me know.  Or I do have the ability to clone our existing instance and troubleshoot.  Thank you, Zach.

Pavel

we got other user complaining same with : GM with Change Volume

it is not about logs, it is about how to handle states (i.e. ok/warning/error)

ok, we will not threat it as error in this situations.

we make a fix during tomorrow and let you know.

dickinsonzach

Wow!  Thank you!

Pavel

looking into that but got a problem:

For flash copy we have these statuses:

* idle_or_copied … normal  (green)
* preparing … warning  (orange) 
* prepared … warning  (orange) 
* copying … normal  (green)
* stopped … error  (red)
* suspended … warning  (orange) 
* stopping … warning  (orange) 

just *stopped status is red.

If we change it to orange for example then flash copy will never be red.

here is data example what we get from storage:

FC Map Level Configuration
------------------------
id,name,source_vdisk_id,source_vdisk_name,target_vdisk_id,target_vdisk_name,group_id,group_name,status,progress,copy_rate,clean_progress,incremental,partner_FC_id,partner_FC_name,restoring,start_time,rc_controlled
1,fcmap1,107,aix_rootvg-migrace,108,aix_rootvg-migrace_01,,,copying,22,0,100,off,,,no,170612150923,no
2,fcmap2,102,aix1_rootvg-migrace,104,aix1_rootvg-migrace_1,,,copying,3,0,100,off,,,no,170616132736,no

Any idea?

dickinsonzach

Good morning, specifically for us we do not use Flash Copy for anything other than Remote Copy.  If we had a way to disable a check, that works for us.

Two ideas:  1.)By default when creating a Remote Copy the Flash Copies get created with the name vdisk#  so vdisk0, vdisk1, etc...  You could ignore Flash Copies that have vdisk# as the target volume.  2.)Can you check to see if a Flash Copy relationship volume is also in a Remote Copy?  3.)OK, three ideas.  Combine 1 and 2.  Ignore if the target is vdisk# AND the relationship exists as a Flash Copy AND a Remote Copy.

I realize there are scenarios that could be overlooked.  Thank you, Zach.

dickinsonzach

Is there an update on this?  We will not be able to upgrade until this is resolved.  Thank you, Zach.

Pavel

sorry, no update yet.

it sounds quite complex, we are waiting for an anwer of our expert.

we will let you know during this week.

dickinsonzach

Sounds good.  Thank you, Zach.

Pavel

Try this fix for that.

Solution is exclude flashes created by global mirroring.

They do not have to be monitored. If anything is wrong with them then global mirror reports problem which we present.

http://www.stor2rrd.com/download-static/svcconfig.pl.gz

Gunzip it and copy to /home/stor2rrd/stor2rrd/bin (755, stor2rrd owner)
-rwxr-xr-x 1 stor2rrd stor2rrd 93479 Oct  3 13:57 bin/svcconfig.pl

If your web browser gunzips it automatically then just rename it: mv svcconfig.pl.gz svcconfig.pl
Assure that file size is same as above listed one

dickinsonzach

1.)Ran the upgrade
2.)Copied the provided file
3.)Rebuilt the GUI:  ./load.sh | tee logs/load.out-initial

At the end of rebuilding the GUI, I got a number of the following errors:
Use of uninitialized value $p_id in substitution (s///) at bin/Storcfg2html.pm line 482, <FCFGR> line 127.

Thoughts?  Thank you, Zach.

Pavel

ignore that.

have disappeared remote flashes from global copy pairs?

It could take 10 - 20 minutes.

dickinsonzach

I will ignore the error from the GUI refresh.

As for the Remote Flashs:

STOR2RRD Copy Services Error for device gcmilton7kmg
name:source_vdisk_name:target_vdisk_name:group_name:status:progress:start_time

 fcmap238:GC_Milton_MG_GCI51001-iASP_FD_RC_43_Aux:GC_Milton_MG_MiltonFC-iASP_FD_FC_43:MILTONFC:stopped:0:

We got one alert, but it had an entry for each Remote Flash.



Thank you, Zach.

Pavel

are global remote flashes still listed in tmp/snapshotting-status/<storage name>.txt ?

dickinsonzach

I have attached the two files in that directory.  They are Remote Partners.  What's interesting is that we 2 other arrays that have a Remote Partnership and they do not appear.  Thank you, Zach.

Pavel

ok, then it works partially right?

on some storages it work and do not some others.

Could you send us data from storage where it still not working?

cd /home/stor2rrd/stor2rrd

3 files : ls -l tmp/<storage name>*txt

https://upload.stor2rrd.com

Thanks

dickinsonzach

Both Remote Copies are not working as I would expect.

The relationship between GCDelta7KMY and GCMilton7KMG is reporting in the GUI, but kicks an unnecessary error each night.

The relationship between GCDeltta7KKA and GCMilton7K7W is not reporting in the GUI.

I have uploaded the requested files for all 4 arrays.  Thank you, Zach.

dickinsonzach

Good morning, any other thoughts on this subject?  Thank you, Zach.

Pavel

Hi, working on it

It is quite complicated to cover all statuses which might appear in copy services.

the first issue has been resolved in previous patch but there were 2 independed issues to be fixed.

We will provide you a fix during today, we already know what need to be done.

dickinsonzach

Sounds good.  Thank you for the quick reply, Zach.

Pavel

Try this cumulative patch since 2.40:

Download link: http://download.stor2rrd.com/patch/stor2rrd-2.40-03.tar

su - stor2rrd
tar xvf stor2rrd-2.40-03.tar
cd stor2rrd-2.40-03
./install.sh

cd /home/stor2rrd/stor2rrd
./load.sh

refresh the GUI when it finishes (Ctrl-F5)

dickinsonzach

I will install the update and let you know how it goes.  Thank you, Zach.

dickinsonzach

I still get:

Use of uninitialized value $p_id in substitution (s///) at bin/Storcfg2html.pm line 482, <FCFGR> line 127.

after re-doing the GUI, but everything else looks good.

Thank you, Zach.

Pavel

ignore that

dickinsonzach

I will ignore that.

It's been over a month, it did pop up about 2 hours ago:

STOR2RRD Copy Services Error for device gcmilton7kmg
name:source_vdisk_name:target_vdisk_name:group_name:status:progress:start_time

 fcmap204:GC_Milton_MG_GCI51001-iASP_FD_RC_9_Aux:GC_Milton_MG_MiltonFC-iASP_FD_FC_9:MILTONFC:stopped:0

Seems to be green now though.  I will post if it happens again.  Thank you, Zach.

dickinsonzach

Good morning, we are seeing STOR2RRD Copy Services Error for device gcmilton7kmg...

fcmap196:GC_Milton_MG_GCI51001-iASP_FD_RC_1_Aux:GC_Milton_MG_MiltonFC-iASP_FD_FC_1:MILTONFC:stopped:0...

Every other day or so and it's showing as an error and paging out.  Is there a way to ignore this error?  Or set to warning so it e-mails, but does not page?

Thank you, Zach.

Pavel

Hi,

is that still the same issue with global mirror?

We do not have evidency of such issues since the last fixes month ago (a fix above in nthe thread).