Slow "comparison/replace" script

CoolBlue · #1 06-22-2014, 01:35 AM

Quote:

Originally Posted by whatsup

I should have said: I can imagine you can calculate the required bytes

That is if you've got a simple object and some insight on how objects are stored in memory.

Hmm. OK, that's too bad...
But anyway, the Byte Comb link you provided helps a little on that.

Quote:

Originally Posted by whatsup

The crash with Mem_Copy oTemp, lPtr, 4 when checking afterwards is still some secret to me.

OK, you don't accept my explanation in post #35?

Well, its only a theory

and I think it crashes before the line

Code:

Set ObjectFromPointer = oTemp

where I thought the problem was...
I also learned since that the first two options below will crash excel...

Code:

Sub crashMem_Copy()
Dim lptr As LongPtr
'    Mem_Copy lptr, ByVal 0, 4      'crashes excel
'    Mem_Copy lptr, ByVal lptr, 4   'crashes excel
    Mem_Copy lptr, lptr, 4            'is fine
End Sub

Quote:

Originally Posted by whatsup

Of course I tried your example with the Class (ever since I trip over your "pionterToSomething" - I should have corrected it but I didn't). What I did change, was "name as string" to "strname as string" - you know why!

Ooops... and... Ooops! two good points.

Quote:

Originally Posted by whatsup

But let's few our new "tool" - we still facing at least two problems:
1. An ordinary object requires only 4 bytes in memory. If I'm not mistaken, every set object to an object that already exists (without using the keyword "New") only is a number in the range of "As Long". But see the example it makes it clear:

Code:

Private Const coBytes As Long = 4
  
 Sub PointsToAnotherObject()
Dim objwks As Worksheet
    Debug.Print
    Debug.Print ObjPtr(Sheet1)
    pionterToSomething "Sheet1  ", ObjPtr(Sheet1), coBytes
Set objwks = Sheet1
    Debug.Print ObjPtr(objwks)
    pionterToSomething "objwks  ", ObjPtr(objwks), coBytes
    
End Sub

So in case you store ObjPtr(objwks) in a variable and check afterwards you might think the referencecount didn't work, because it's still there (unless you remove Sheet1 from your workbook). So there won't be any hint whether or not the 4 Bytes are freed, because afterwards you always will find the object of origin.

No problem, you can use VarPtr instead of ObjPtr and you will get the reference address that is stored in the local pointer. Your local variable objwks, is of course just a pointer with a reference to the worksheet object. You can see how this works here...

Code:

Sub getReference()
#If Win64 Then
    Const PTR_LENGTH As Long = 8
#Else
    Const PTR_LENGTH As Long = 4
#End If
Const coBytes As Long = 4
Dim objwks As Worksheet, lptr As LongPtr, objName As String, objwksPtr As LongPtr

    Debug.Print
'   Sheet1 Object reference and first coBytes bytes
    lptr = objPtr(Sheets(1)): objName = "Sheet1 "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
    
'   Using objPtr, can't diferentiate because the reference is resolved before reporting
    Set objwks = Sheets(1)
    lptr = objPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
        
'   Using varPtr, can diferentiate.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr"
    
'   The contents of the local variable (objwks) is the address of the global object but byte-reversed (little endian)
'   Load the reference into a LongPtr type and read back the byte-corrected (big endian) value of the reference
    Mem_Copy objwksPtr, ByVal VarPtr(objwks), PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)

'   Can also do this passing objwks as ByRef (default)
    Mem_Copy objwksPtr, objwks, PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)

'   Or like this...
    Mem_Copy ByVal VarPtr(objwksPtr), objwks, PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)
    
End Sub

Output:

Code:

Sheet1  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x002AEF90 : Contents: 0x78052418    using varPtr
objwks  points to: 0x18240578
objwks  points to: 0x18240578
objwks  points to: 0x18240578

There are also some examples of how to call Mem_Copy (RtlMoveMemory) that might help to understand why it crashes.

And here you can see how the local variable's value is zero when its set to nothing...

Code:

Sub getReference2()
#If Win64 Then
    Const PTR_LENGTH As Long = 8
#Else
    Const PTR_LENGTH As Long = 4
#End If
Const coBytes As Long = 4
Dim objwks As Worksheet, lptr As LongPtr, objName As String, objwksPtr As LongPtr

    Debug.Print
'   Sheet1 Object reference and first coBytes bytes
    lptr = objPtr(Sheets(1)): objName = "Sheet1 "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
    
'   Using varPtr, can see the local variable is set to nothing.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr, set to Nothing"
    
'   Using objPtr, can't diferentiate because the reference is resolved before reporting
    Set objwks = Sheets(1)
    lptr = objPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
        
'   Using varPtr, can diferentiate.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr"
    
'   The contents of the local variable (objwks) is the address of the global object but byte-reversed (little endian)
'   Load the reference into a LongPtr type and read back the byte-corrected (big endian) value of the reference
    Mem_Copy objwksPtr, ByVal VarPtr(objwks), PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)
    
End Sub

Output:

Code:

Sheet1  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x002AEF90 : Contents: 0x00000000    using varPtr, set to Nothing
objwks  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x002AEF90 : Contents: 0x78052418    using varPtr
objwks  points to: 0x18240578

Quote:

Originally Posted by whatsup

2. If memory is freed, we might find "cleared to zero", but there might be something else assigned to the pointer. Up to now it only occured when testing with pointers to a Long-variable, but I think it always can happen. Therefore we better compare - keeping in mind point 1

If the memory is set to zero or overwritten, it makes no difference; it is still clear evidence that the variable's memory space has been released. And that's been my point all along...

In summary, I cant see any reason to set local objects to nothing at the end of a sub. Even if there is an error and the sub doesn't complete normally, or if there are structures that fool the reference counting, everything is still cleaned up in the former and the set to nothing does nothing extra in the latter.

I learned a lot from this exercise and added some nice routines to my library, so this has been most interesting!

For anyone who wants to run the above routines, here is a .bas file with everything you need... well its .txt but just need to change it to .bas and you can import into a module.

whatsup · #2 06-22-2014, 06:05 PM

Sorry for the delay, but i kept thinking and thinking ...

Point1: Post #35
I basically share your opinion about the application crash on the attempt to Mem_copy something what doesn't match a 4 Bytes Code.
By accident I choose first a range-object (and experienced the crashes), afterwards I changed to a worksheet-object (avoiding crashes I didn't any mem-copy to an object or a variant). I was just looking at the numbers of ObjPtr which are always the same whether you obtain them directly from the object
ObjPtr(Sheet1)
or from the variable
ObjPtr(objwks)
Therefore I wondered, why did excel crash, because even with the variable gone I ask at the end of the Copy of ObjPtr(Range("A1")), and since the Range still exists and hasn't changed, what's the problem to copy it?
It took me a long time to figure out there is a difference between Range and Worksheet:
Whereas Worksheet keeps his pointer throughout the application, this isn't the case with Range. A Range changes its pointer, furthermore the variable (objRange) set to the Range is assigned another pointer.

I said the choice was by accident but actually I'm glad about the choice because otherwise it would have led in a complete different direction...

Point2:
I regret the lack of knowlegde about the architecture of objects and what they look like in memory. I really do, probably it would make things a lot easier knowing about this things.

Point3:

Quote:

If the memory is set to zero or overwritten, it makes no difference; it is still clear evidence that the variable's memory space has been released. And that's been my point all along...

I would feel more comfortable if evidence would present itself more obvious, for example in just showing only zeros (as it does with a simple class). Why can't excel give a pointer some rest after he did a job

But Yes, I agree, at least for the moment let's depend on the change.

Point4:
According to Point 4, I agree, memory is freed whether or not the variable is set to Nothing (at least valid for the macros I tried up to now). A surprise to me: Classes are destroyed without explicity destroying them.
But I don't agree on this

Quote:

...or if there are structures that fool the reference counting...

If you refer to the example with circular references within classes, that's one thing which the example establishes most clearly: None of them get destroyed.

Attached the file including 4 tests - as well the CircRef - with some kind of summary.

CoolBlue · #3 06-22-2014, 07:37 PM

Quote:

Originally Posted by whatsup

Sorry for the delay, but i kept thinking and thinking ...

Point1: Post #35
I basically share your opinion about the application crash on the attempt to Mem_copy something what doesn't match a 4 Bytes Code.
By accident I choose first a range-object (and experienced the crashes), afterwards I changed to a worksheet-object (avoiding crashes I didn't any mem-copy to an object or a variant). I was just looking at the numbers of ObjPtr which are always the same whether you obtain them directly from the object
ObjPtr(Sheet1)
or from the variable
ObjPtr(objwks)
Therefore I wondered, why did excel crash, because even with the variable gone I ask at the end of the Copy of ObjPtr(Range("A1")), and since the Range still exists and hasn't changed, what's the problem to copy it?
It took me a long time to figure out there is a difference between Range and Worksheet:
Whereas Worksheet keeps his pointer throughout the application, this isn't the case with Range. A Range changes its pointer, furthermore the variable (objRange) set to the Range is assigned another pointer.

I said the choice was by accident but actually I'm glad about the choice because otherwise it would have led in a complete different direction...

I'll have to take some time to think about that, but they are interesting observations...
I'm surprised that the crashMem_Copy routine in my previous post crashes excel because the destination pointer is fine. Why crash when the source pointer is set to zero? Its only a read... Any thoughts?

Quote:

Originally Posted by whatsup

I regret the lack of knowlegde about the architecture of objects and what they look like in memory. I really do, probably it would make things a lot easier knowing about this things.

Well, clearly you are not alone! Nobody seems to have any knowledge about this, at least I cant find any.
But your link was very helpful, I've seen the byte comb site before but it was good to be reminded of it, it informed my thinking on this.

Quote:

Originally Posted by whatsup

I would feel more comfortable if evidence would present itself more obvious, for example in just showing only zeros (as it does with a simple class). Why can't excel give a pointer some rest after he did a job

But Yes, I agree, at least for the moment let's depend on the change.

It seems to me that the Erase Method in VBS's "garbage collection" system does indeed set the first 8 bytes to zeros. The point is, after a crash and re-start of another routine, a LOT has changed and the space will surely have been used for something else. So the zeros have been over-written.

Quote:

Originally Posted by whatsup

According to Point 4, I agree, memory is freed whether or not the variable is set to Nothing (at least valid for the macros I tried up to now). A surprise to me: Classes are destroyed without explicity destroying them.

Yes, I saw from one of your early quotes that you would be surprised about the classes being destroyed and I wanted to give you clear evidence about that and that's why I did the class example.
Its quite simple really, all object life cycle is managed by reference counting: when the reference count goes to zero, the object is scheduled for deletion. There are no no exceptions.
I guess this will be a background process that is run when VBARuntime has free time and it will just scan the reference count field in the object table and "Erase" those with zero reference. The ERASE method will not delete the object from the object table, but it will release any memory used by the object's structure.

Quote:

Originally Posted by whatsup

But I don't agree on this
If you refer to the example with circular references within classes, that's one thing which the example establishes most clearly: None of them get destroyed.

Attached the file including 4 tests - as well the CircRef - with some kind of summary.

OK, now we get to the main point

The full quote on this is:

Quote:

Originally Posted by CoolBlue

In summary, I cant see any reason to set local objects to nothing at the end of a sub. Even if there is an error and the sub doesn't complete normally, or if there are structures that fool the reference counting, everything is still cleaned up in the former and the set to nothing does nothing extra in the latter.

Sorry, the language is a bit unclear so I've added emphasis...

I am saying that setting to nothing does nothing extra in these cases. Im saying it makes no difference. The problem will still be there.
I am arguing against using set to nothing remember?

As you can see in your example, the result after run-time is the same if you have

Code:

Set objcls1 = Nothing

Or

Code:

'Set objcls1 = Nothing

THAT is my point: you cannot fix these issues with set to nothing. One of the links offered by my friend @macropd as a reason to use set to nothing was referring to these structures and my point was and still is that set to nothing makes no difference.

So, the two points I was trying to argue against are:

Your second point in post#7

Quote:

Originally Posted by whatsup

- The main reason: There are different types of objects, for instance classes you must unload manually; if creating a new instance of an object (whenever using the keyword "New") you have to unload it manually as well...

The claim by @macropod in post#7 that this link was supporting the use of set to nothing at the end of a sub as a way to manage (for example) objects with circular references.

I think we do agree on those...

And thanks for the attachment... your documentation style is much nicer than mine, I will study it and improve my ways!

whatsup · #4 06-22-2014, 10:15 PM

Quote:

In summary, I cant see any reason to set local objects to nothing at the end of a sub. Even if there is an error and the sub doesn't complete normally, or if there are structures that fool the reference counting, everything is still cleaned up in the former and the set to nothing does nothing extra in the latter.

Ok, that way I agree. Still, if short of memory you can deallocate memory earlier using Set to Nothing. But otherwise it's done automatically when leaving the sub. That's shown nicely in Run11, when you skip the part Set to nothing, the Terminate-Event of the class is still triggered (but I see you mentioned this already in your post)

And I remember very well that you argue against - therefore we are here

But I enjoy. That way I learn quite a lot about memory, which really is due in regard that obviously MS has improved on the subject.

Yeah, the VarPtr of an object - honestly I don't know if it makes sense. I tried to figure out whether the Mem_ReadHex of VarPtr points to the ObjPtr by converting the HexToDec, but was disapointed by the result. Even worse HexToDec comes up with negative figures, so what shall we make of this? Maybe Val("&H"&...) isn't the right tool?

Quote:

Why crash when the source pointer is set to zero? Its only a read... Any thoughts?

I don't know, maybe because it doesn't exist a pointer "Zero" within memory? Maybe "Zero" makes a program turn off the track ways before, but we are interfering far behind the curtain where nothing is provided anymore to reject the attempt? In fact you cannot copy anything if the pointer is Zero, therefore I just excluded it from any attempt.

Documentation is a selfish thing - I know after several days I will end up with x files not knowing anymore what I intended to achieve, just testing, playing, ...

But one thing is sure, the time I used formatting the immediate window, I rather would have spent for another output. But for the moment it's ok.

Me as well, I spent a lot of time searching the web for information about "memory" and related words, but google misunderstands what I'm looking for, and mostly comes up with things I'm not interested in.

CoolBlue · #5 06-23-2014, 02:38 AM

Quote:

Originally Posted by whatsup

Ok, that way I agree. Still, if short of memory you can deallocate memory earlier using Set to Nothing. But otherwise it's done automatically when leaving the sub. That's shown nicely in Run11, when you skip the part Set to nothing, the Terminate-Event of the class is still triggered (but I see you mentioned this already in your post)

OK, cool

And yes, I agree, its a good idea to set objects to nothing earlier if you don't need them anymore... but if you are already at the end of the sub then... yep, we agree on that.

Quote:

Originally Posted by whatsup

And I remember very well that you argue against - therefore we are here

But I enjoy. That way I learn quite a lot about memory, which really is due in regard that obviously MS has improved on the subject.

I am enjoying the learning also!

Quote:

Originally Posted by whatsup

Yeah, the VarPtr of an object - honestly I don't know if it makes sense. I tried to figure out whether the Mem_ReadHex of VarPtr points to the ObjPtr by converting the HexToDec, but was disapointed by the result. Even worse HexToDec comes up with negative figures, so what shall we make of this? Maybe Val("&H"&...) isn't the right tool?

Ah, this one I can explain...
The memory image is in a format called little endian. Intel-based systems store bytes little end first so the least significant byte is on the left. You have to flip the bytes to see the true hex value...
If you look at the last section of getReference2() in my previous post you will see how that I move the contents of the local pointer (the address of Sheet1) into another LongPtr type called objwksPtr. I can then print out the hex value of that pointer and VBA will interpret it correctly.

Code:

01 Sheet1  : Address: 0x1D6DA618 : Contents: 0xC0EA4C18    using objPtr
02 objwks  : Address: 0x002AEF90 : Contents: 0x00000000    using varPtr, set to Nothing
03 objwks  : Address: 0x1D6DA618 : Contents: 0xC0EA4C18    using objPtr
04 objwks  : Address: 0x002AEF90 : Contents: 0x18A66D1D    using varPtr
05 objwks  points to: 0x1D6DA618

Line 04 is from this...
'   Using varPtr, can diferentiate.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr"
It produces this image of the contents of the objwks reference
    0x18A66D1D

Line 05 is from this
'   The contents of the local variable (objwks) is the address of the global object but byte-reversed (little endian)
'   Load the reference into a LongPtr type and read back the byte-corrected (big endian) value of the reference
    Mem_Copy objwksPtr, ByVal VarPtr(objwks), PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)
It produces this image of the contents of the objwks reference
    0x1D6DA618
And this is the correct address of the Sheet1 Object

But in fact they are the same number but in different formats...
    1D6DA618 18A66D1D

I just did it like that for illustration, the other way is to shuffle the bytes in the Mem_ReadHex routine. I added a sub called Mem_ReadHex_Endian_Aware to your Mod00_PublicFunctionsAPI module and incorperated it into your pointerToSomething routine in the attached version of your spreadsheet.
Ive also attached a couple of pics to try to make it clear...

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Wierd "script code" in a downloaded .doc file	CNBarnes	Word	2	10-18-2012 02:07 AM
replace data from variable with "sub and super script" from excel to word by vba	krishnaoptif	Word VBA	9	06-22-2012 05:08 AM
How to choose a "List" for certain "Heading" from "Modify" tool?	Jamal NUMAN	Word	2	07-03-2011 03:11 AM
Rules and Alerts: "run a script"?	discountvc	Outlook	0	06-15-2010 07:36 AM
An "error has occurred in the script on this page"	decann	Outlook	8	09-03-2009 08:54 AM