Slow "comparison/replace" script

CoolBlue · #31 06-20-2014, 02:57 AM

Quote:

Originally Posted by whatsup

Code:

Public Declare PtrSafe Sub Mem_Copy Lib "kernel32" Alias "RtlMoveMemory" ( _
    ByRef Destination As Any, _
    ByRef Source As Any, _
    ByVal Length As Long)
  
' Platform-independent method to return the full zero-padded
' hexadecimal representation of a pointer value
Function HexPtr(ByVal Ptr As LongPtr) As String
    HexPtr = Hex$(Ptr)
    HexPtr = String$((PTR_LENGTH * 2) - Len(HexPtr), "0") & HexPtr
End Function
 
Public Function Mem_ReadHex(ByVal Ptr As LongPtr, ByVal Length As Long) As String
    Dim bBuffer() As Byte, strBytes() As String, i As Long, ub As Long, b As Byte
    ub = Length - 1
    ReDim bBuffer(ub)
    ReDim strBytes(ub)
    Mem_Copy bBuffer(0), ByVal Ptr, Length
    For i = 0 To ub
        b = bBuffer(i)
        strBytes(i) = IIf(b < 16, "0", "") & Hex$(b)
    Next
    Mem_ReadHex = Join(strBytes, "")
End Function
 
Sub ExampleForScalar()
 
Dim lngValue As Long
     
lngValue = 100
    
lng_VarPtr = VarPtr(lngValue)
    
    Debug.Print "lngValue : 0x"; HexPtr(lng_VarPtr); _
                 " : 0x"; Mem_ReadHex(lng_VarPtr, 4)
      
End Sub
 
Sub ValueIsGone()
 Debug.Print "lngValue : 0x"; HexPtr(lng_VarPtr); _
                 " : 0x"; Mem_ReadHex(lng_VarPtr, 4)
End Sub

In this example with ValueIsGone() you track down the pointer obtained from ExampleForScalar() and the reading of Memory says the value is zero.

Now with objects it isn't that easy, and that's where I'm struggling. I trying to recover the object from memory by it's pointer by adding this code:

Code:

Function ObjectFromPointer(lPtr As Long) As Object
     Dim oTemp As Object
     
     Mem_Copy oTemp, lPtr, 4
     Set ObjectFromPointer = oTemp
End Function
  
Sub ExampleForObject()
Dim objRange As Object
     
Set objRange = Sheets(1).Range("A1")
    
lng_ObjPtr = ObjPtr(objRange)
                     
'Set objRange = Nothing
    
End Sub
  
Sub ObjectIsGone()
Dim objRecover As Object
  
     Set objRecover = ObjectFromPointer(lng_ObjPtr)
  
End Sub

But Mem_Copy in the function ObjectFromPointer() is killing the excel application, no debugging but killing, so be careful with that one.
I'm assuming that the pointer maybe doesn't exist anymore, which would be fine, indicating that the object was taken from memory, but we can have that proof with excel going to hell.

Any idea on this?

Yes, the scalar routine (Mem_ReadHex) has a Length parameter that is used to politely reserve bytes for Mem_Copy to blast into. Your object routine has no such parameter. The Mem_Copy routine in ObjectFromPointer is just writing from the base of and Object structure and destroying the VBA Object structure.then you set objRecover to an irregular structure and that's why it crashes.
In order for it to work, you need to figure out the structure of the range object in your example and reserve a buffer with sufficient bytes, for Mem_Copy to dump into.
You need to create a 4 byte buffer just like in the scalar routine to receive the bytes you transfer.

CoolBlue · #32 06-20-2014, 03:49 AM

Try this instead...

Code:

Option Explicit
Public lng_VarPtr As Long
Public lng_ObjPtr As Long
  
 #If Win64 Then
    Public Const PTR_LENGTH As Long = 8
#Else
    Public Const PTR_LENGTH As Long = 4
#End If
 
Public Declare PtrSafe Sub Mem_Copy Lib "kernel32" Alias "RtlMoveMemory" ( _
    ByRef Destination As Any, _
    ByRef Source As Any, _
    ByVal Length As Long)
  
' Platform-independent method to return the full zero-padded
' hexadecimal representation of a pointer value
Function HexPtr(ByVal Ptr As LongPtr) As String
    HexPtr = Hex$(Ptr)
    HexPtr = String$((PTR_LENGTH * 2) - Len(HexPtr), "0") & HexPtr
End Function
 
Public Function Mem_ReadHex(ByVal Ptr As LongPtr, ByVal Length As Long) As String
    Dim bBuffer() As Byte, strBytes() As String, i As Long, ub As Long, b As Byte
    ub = Length - 1
    ReDim bBuffer(ub)
    ReDim strBytes(ub)
    Mem_Copy bBuffer(0), ByVal Ptr, Length
    For i = 0 To ub
        b = bBuffer(i)
        strBytes(i) = IIf(b < 16, "0", "") & Hex$(b)
    Next
    Mem_ReadHex = Join(strBytes, "")
End Function
 
Sub pionterToSomething(name As String, lng_Ptr As Long, Length As Long)
    Debug.Print name & " : 0x"; HexPtr(lng_Ptr); _
                                     " : 0x"; Mem_ReadHex(lng_Ptr, Length)
End Sub
Sub ExampleForScalar()
 
Dim lngValue As Long
    lngValue = &HAAAAAAAA
    lng_VarPtr = VarPtr(lngValue)
    pionterToSomething "lngValue", lng_VarPtr, 4
End Sub
   
Sub ExampleForObject()
Dim objRange As Object

    Set objRange = ActiveCell
    lng_ObjPtr = ObjPtr(objRange)                    
    pionterToSomething "objRange", lng_ObjPtr, 4

'Set objRange = Nothing
End Sub
Sub ObjectIsGone()
    Debug.Print
    ExampleForObject
    pionterToSomething "objRange", lng_ObjPtr, 4
End Sub
Sub ValueIsGone2()
    Debug.Print
    ExampleForScalar
    pionterToSomething "lngValue", lng_VarPtr, 4
End Sub

CoolBlue · #33 06-20-2014, 06:36 AM

Maybe this will do it for you...

Create a Custom Class

Code:

'Class2
Option Explicit

Public i As Long
Public j As Long
Public k As Long

Then add to the standard Module with other XXXIsGone code

Code:

Const clsBytes As Long = 32 * 4
'///////////////////////////////////




'///////////////////////////////////
Sub ExampleForClass()
Dim objCls As Class2
Dim str As String
     
    Set objCls = New Class2
    objCls.i = &HAAAAAA
    objCls.j = &HABABAB
    objCls.k = &HBCBCBC
    
    lng_ObjPtr = ObjPtr(objCls)
                     
    str = PionterToSomething("objCls", lng_ObjPtr, clsBytes)
    Debug.Print str

    'Set objCls = Nothing
End Sub
'///////////////////////////////////


'///////////////////////////////////
Sub classIsGone()
    Debug.Print
    ExampleForClass
    DoEvents
    printPionterToSomething "objCls", lng_ObjPtr, clsBytes
End Sub

And try with and without the set to nothing...
No difference.

whatsup · #34 06-20-2014, 10:55 AM

Hi CoolBlue

Quote:

In order for it to work as, you need to figure out the structure of the range object in your example and reserve a buffer with sufficient bytes, for Mem_Copy to dump into.

Yes that's exactly the problem. I did try the same way as you showed, and in case of your last example it can be done, because it's simple and you can calculate the required bytes. But usually it's unpredictable the requirement of an object to copy, and therefore it won't work, - that way.

Now I gave some more thoughts to the function ObjectFromPointer(). That's because, what do I gain by copying the contents assigned to a pointer? Sure if still pointing to the same object (as in the example with objRange) I can ask if "Nothing" or use a property of the initial object to identify it. But in case the pointer is already assigned something else it will give me a hard time to keep the macro out of trouble.
Therefore I wonder if it would make sense to read always a fix parameter of length for example 32 and just compare the results. If they show a difference, it will tell that something else got assigned to the pointer, and that's good enough.

Just to avoid missunderstanding: I didn't say, that the reference count doesn't work at all. It's just as Bob put it "there might be circumstances when it doesn't".
In the moment we do with simple structures because it's about figuring out a way to keep track on ongoing things within memory. Therefore I'm expecting reference count doing allright and would be dissapointed if failing (though creating a new instance of an object was said in former times might be already to much for automatic deallocation, the same as for not handled errors).

CoolBlue · #35 06-20-2014, 04:17 PM

Quote:

Originally Posted by whatsup

Hi CoolBlue

Yes that's exactly the problem. I did try the same way as you showed, and in case of your last example it can be done, because it's simple and you can calculate the required bytes. But usually it's unpredictable the requirement of an object to copy, and therefore it won't work, - that way.

Yep, I assume that the Any Type of the parameters of your MEM_Copy declaration will typecast any Object as a pointer to that object, so

Code:

Mem_Copy oTemp, lPtr, 4

Is like

Code:

Mem_Copy ObjPtr(oTemp), lPtr, 4

The problem is that you are overwriting the first 4 bytes (in this case) of an Object type variable structure with the first 4 bytes of a Range type variable structure, coz IPtr is a pointer to the base address of a Range Object.
For sure, this will corrupt the structure of oTemp.
Then when you do

Code:

Set ObjectFromPointer = oTemp

Excel will crash coz it doesn't find what it needs at the base address of oTemp because its corrupted already.

But I'm interested that you say you said you can calculate the required bytes... How do you do that? I've been searching for an explanation of the byte structure that VBA uses for storing it's various types: do you have that?

Quote:

Originally Posted by whatsup

Now I gave some more thoughts to the function ObjectFromPointer(). That's because, what do I gain by copying the contents assigned to a pointer? Sure if still pointing to the same object (as in the example with objRange) I can ask if "Nothing" or use a property of the initial object to identify it. But in case the pointer is already assigned something else it will give me a hard time to keep the macro out of trouble.
Therefore I wonder if it would make sense to read always a fix parameter of length for example 32 and just compare the results. If they show a difference, it will tell that something else got assigned to the pointer, and that's good enough. .

Yep, I had the same thought. Did you try my example with a Class Object? I arbitrarily set the bytes count to 32*4 just to get a look at the structure, to search for the value I set the properties to (hex AAAAAA, or something like that) and you can see it clearly, a little way into the buffer. Try it and you will see what I mean.
But anyway, it seems like very start of that structure is always set to zeros when the variable is erased by VBA's memory management. So you can always see that the object has been erased. This is the best indication I've seen so far to show the life cycle of the variable.
Here are a few sample runs...

Code:

objCls : 0x18E11A58 : 0x04C0031500000000741AE1181C753904047539040000000000000000ECEC831A0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00F4E9101500000000000000009C0601005FB4AF2E060100802300E7719425E771
objCls : 0x18E11A58 : 0x0000000000000000741AE11800000000000000000000000000000000ECEC831A0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00F4E9101500000000000000009C0601005FB4AF2E060100800D00530041004100

objCls : 0x18E11A58 : 0x04C0031500000000741AE1181C753904047539040000000000000000ECEC831A0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00F4E9101500000000000000009C0601005FB4AF2E060100802300E7719425E771
objCls : 0x18E11A58 : 0x0000000000000000741AE11800000000000000000000000000000000ECEC831A0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00F4E9101500000000000000009C0601005FB4AF2E060100802300E7719425E771

objCls : 0x18E11A58 : 0x04C0031500000000741AE1181C753904047539040000000000000000ECEC831A0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00F4E9101500000000000000009C0601005FB4AF2E060100804400E7719425E771
objCls : 0x18E11A58 : 0x0000000000000000741AE11800000000000000000000000000000000ECEC831A0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00F4E9101500000000000000009C0601005FB4AF2E060100801800E7719425E771

And he you can see the reference counting doing its job...

Code:

objCls : 0x18E11950 : 0xEC2C0915000000006C19E118648948044C8948040000000000000000ECEC831A0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC000000000004E9101500000000000000003EB4AF2EE50000880000000000000000
after exit...
objCls : 0x18E11950 : 0x00000000000000006C19E11800000000000000000000000000000000ECEC831A0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC000000000004E9101500000000000000003EB4AF2EE50000880000000000000000

objCls : 0x18E11950 : 0xEC2C0915000000006C19E118648948044C8948040000000000000000ECEC831A0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC000000000004E9101500000000000000003EB4AF2EE50000880000000000000000
Set to Nothing...
objCls : 0x18E11950 : 0x00000000000000006C19E11800000000000000000000000000000000ECEC831A0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC000000000004E9101500000000000000003EB4AF2EE50000880000000000000000
after exit...
objCls : 0x18E11950 : 0x00000000000000006C19E11800000000000000000000000000000000ECEC831A0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC000000000004E9101500000000000000003EB4AF2EE50000880000000000000000

As you can see, the first 8 bytes is always cleared to zeros.

Quote:

Originally Posted by whatsup

Just to avoid misunderstanding: I didn't say, that the reference count doesn't work at all. It's just as Bob put it "there might be circumstances when it doesn't".
In the moment we do with simple structures because it's about figuring out a way to keep track on ongoing things within memory. Therefore I'm expecting reference count doing allright and would be dissapointed if failing (though creating a new instance of an object was said in former times might be already to much for automatic deallocation, the same as for not handled errors).

AH, OK, fair enough. Yep, sure. Well I think we have a detection tool that works now, to a reasonable level of confidence, so the next thing I was thinking to do is to create some structures that will not be released - like a Class with a circular reference for example - and see how that behaves. I'll also try crashing out of the sub on an error to see if that doesn't release the variable as well.

CoolBlue · #36 06-20-2014, 06:45 PM

Here are the results for a circular list compared to a single object...

Single Object

Code:

No Cleanup, No Error
SingleObject set...
objCls : 0x18DECA08 : 0x64C67D1C0000000024CADE18EC577803D45778030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000
after exit...
objCls : 0x18DECA08 : 0x000000000000000024CADE1800000000000000000000000000000000ECECAE1B0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000

With Cleanup, No Error
SingleObject set...
objCls : 0x18DECA08 : 0x64C67D1C0000000024CADE18EC577803D45778030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000
Set to Nothing...
objCls : 0x18DECA08 : 0x000000000000000024CADE1800000000000000000000000000000000ECECAE1B0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000
after exit...
objCls : 0x18DECA08 : 0x000000000000000024CADE1800000000000000000000000000000000ECECAE1B0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000

With Cleanup, Error after setting the pointer
SingleObject set...
objCls : 0x18DECA08 : 0x64C67D1C0000000024CADE18EC577803D45778030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000
Set to Nothing...
objCls : 0x18DECA08 : 0x000000000000000024CADE1800000000000000000000000000000000ECECAE1B0100000000000000000000006E1C000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000
After Error, Before End Sub
after error...
objCls : 0x18DECA08 : 0x8841BF182AA30000712CBB1BDC25BB1BC6D3B71B10000400F88656080000000000000000000000000000000006000000100000005029881C000000005F005F005300520050005F000000000000000000FD3D013B6500008C0000000028000000

No Cleanup, Error after setting the pointer
SingleObject set...
objCls : 0x18DECA08 : 0x64C67D1C0000000024CADE18B45778039C5778030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00000000002489C3180000000000000000FD3D013B6500008C0000000028000000
After Error, Before End Sub
after error...
objCls : 0x18DECA08 : 0x8841BF182AA30000712CBB1BDC25BB1BC6D3B71B1000040030865608000000000000000000000000000000000600000010000000D82B881C000000005F005F005300520050005F000000000000000000FD3D013B6500008C0000000028000000

Circular List

Code:

No Cleanup, No Error
circularList set...
objCls : 0x18DECA08 : 0x64C67D1C0000000024CADE18EC577803D45778030000000000000000ECECAE1B0200000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00C8CC82182489C3180000000000000000FD3D013B6500008C0000000028000000
after exit...
objCls : 0x18DECA08 : 0x64C67D1C0000000024CADE18EC577803D45778030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC00C8CC82182489C3180000000000000000FD3D013B6500008C0000000028000000

With Cleanup, No Error
circularList set...
objCls : 0x0DA0DC70 : 0x64C67D1CC8CC82188CDCA00D245878030C5878030000000000000000ECECAE1B0200000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC0000A9B70D2489C31800000000000000005E55B53900000088E073AF1B08179D0D
Set to Nothing...
objCls : 0x0DA0DC70 : 0x64C67D1CC8CC82188CDCA00D245878030C5878030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC0000A9B70D2489C31800000000000000005E55B53900000088E073AF1B08179D0D
after exit...
objCls : 0x0DA0DC70 : 0x64C67D1CC8CC82188CDCA00D245878030C5878030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC0000A9B70D2489C31800000000000000005E55B53900000088E073AF1B08179D0D

With Cleanup, Error after setting the pointer
circularList set...
objCls : 0x1882CDD0 : 0x64C67D1C00A9B70DECCD8218945878037C5878030000000000000000ECECAE1B0200000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC0078CD82182489C318000000000000000042AC103B000000804400FA171A000000
Set to Nothing...
objCls : 0x1882CDD0 : 0x64C67D1C00A9B70DECCD8218945878037C5878030000000000000000ECECAE1B0100000000000000000000000F10000000000000AAAAAA00ABABAB00BCBCBC0078CD82182489C318000000000000000042AC103B000000804400FA171A000000
After Error, Before End Sub
after error...
objCls : 0x1882CDD0 : 0x0D00E7719425E7717825E7715825E7714825E771000000006890851868908518000000000000000078908518000000000900000000000000000000000000000001000000DCCD821800000000FA050E0042AC103B000000802E00E7719425E771

I think we've found the higgs boson!
The detector works...

You have to scroll across a bit but you can see that the object is not released when there is a circular reference. Its not released by VBA or by the set to nothing.

whatsup · #37 06-20-2014, 07:13 PM

I should have said: I can imagine you can calculate the required bytes

That is if you've got a simple object and some insight on how objects are stored in memory.

But forget about this remark, I hope we don't have to go down this road all the way.

The crash with Mem_Copy oTemp, lPtr, 4 when checking afterwards is still some secret to me. (though I might have an idea where it comes from, but I might be wrong so I will put this aside for the moment, furthermore it's not important in the moment).

Of course I tried your example with the Class (ever since I trip over your "pionterToSomething" - I should have corrected it but I didn't). What I did change, was "name as string" to "strname as string" - you know why!
And, yes, I'm familiar as to the results showing.

But let's few our new "tool" - we still facing at least two problems:
1. An ordinary object requires only 4 bytes in memory. If I'm not mistaken, every set object to an object that already exists (without using the keyword "New") only is a number in the range of "As Long". But see the example it makes it clear:

Code:

Private Const coBytes As Long = 4
  
 Sub PointsToAnotherObject()
Dim objwks As Worksheet
    Debug.Print
    Debug.Print ObjPtr(Sheet1)
    pionterToSomething "Sheet1  ", ObjPtr(Sheet1), coBytes
Set objwks = Sheet1
    Debug.Print ObjPtr(objwks)
    pionterToSomething "objwks  ", ObjPtr(objwks), coBytes
    
End Sub

So in case you store ObjPtr(objwks) in a variable and check afterwards you might think the referencecount didn't work, because it's still there (unless you remove Sheet1 from your workbook). So there won't be any hint whether or not the 4 Bytes are freed, because afterwards you always will find the object of origin.
2. If memory is freed, we might find "cleared to zero", but there might be something else assigned to the pointer. Up to now it only occured when testing with pointers to a Long-variable, but I think it always can happen. Therefore we better compare - keeping in mind point 1

Just got the message of your new post, I will have a look at it now. But I anyway wanted to say, circular reference in classes - I don't know if you can free memory at all - without closing excel.

whatsup · #38 06-20-2014, 08:20 PM

The SingleObject example:
In the error-cases the pointer is assigned to something else. By the look of it, I tend to say the error-window is stored there since you twice got the same hex-code - is this possible?.

Concerning the Circular List, well I don't know what macro is behind. But for the error it's again an error-window?

CoolBlue · #39 06-22-2014, 01:35 AM

Quote:

Originally Posted by whatsup

I should have said: I can imagine you can calculate the required bytes

That is if you've got a simple object and some insight on how objects are stored in memory.

Hmm. OK, that's too bad...
But anyway, the Byte Comb link you provided helps a little on that.

Quote:

Originally Posted by whatsup

The crash with Mem_Copy oTemp, lPtr, 4 when checking afterwards is still some secret to me.

OK, you don't accept my explanation in post #35?
Well, its only a theory

and I think it crashes before the line

Code:

Set ObjectFromPointer = oTemp

where I thought the problem was...
I also learned since that the first two options below will crash excel...

Code:

Sub crashMem_Copy()
Dim lptr As LongPtr
'    Mem_Copy lptr, ByVal 0, 4      'crashes excel
'    Mem_Copy lptr, ByVal lptr, 4   'crashes excel
    Mem_Copy lptr, lptr, 4            'is fine
End Sub

Quote:

Originally Posted by whatsup

Of course I tried your example with the Class (ever since I trip over your "pionterToSomething" - I should have corrected it but I didn't). What I did change, was "name as string" to "strname as string" - you know why!

Ooops... and... Ooops! two good points.

Quote:

Originally Posted by whatsup

But let's few our new "tool" - we still facing at least two problems:
1. An ordinary object requires only 4 bytes in memory. If I'm not mistaken, every set object to an object that already exists (without using the keyword "New") only is a number in the range of "As Long". But see the example it makes it clear:

Code:

Private Const coBytes As Long = 4
  
 Sub PointsToAnotherObject()
Dim objwks As Worksheet
    Debug.Print
    Debug.Print ObjPtr(Sheet1)
    pionterToSomething "Sheet1  ", ObjPtr(Sheet1), coBytes
Set objwks = Sheet1
    Debug.Print ObjPtr(objwks)
    pionterToSomething "objwks  ", ObjPtr(objwks), coBytes
    
End Sub

So in case you store ObjPtr(objwks) in a variable and check afterwards you might think the referencecount didn't work, because it's still there (unless you remove Sheet1 from your workbook). So there won't be any hint whether or not the 4 Bytes are freed, because afterwards you always will find the object of origin.

No problem, you can use VarPtr instead of ObjPtr and you will get the reference address that is stored in the local pointer. Your local variable objwks, is of course just a pointer with a reference to the worksheet object. You can see how this works here...

Code:

Sub getReference()
#If Win64 Then
    Const PTR_LENGTH As Long = 8
#Else
    Const PTR_LENGTH As Long = 4
#End If
Const coBytes As Long = 4
Dim objwks As Worksheet, lptr As LongPtr, objName As String, objwksPtr As LongPtr

    Debug.Print
'   Sheet1 Object reference and first coBytes bytes
    lptr = objPtr(Sheets(1)): objName = "Sheet1 "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
    
'   Using objPtr, can't diferentiate because the reference is resolved before reporting
    Set objwks = Sheets(1)
    lptr = objPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
        
'   Using varPtr, can diferentiate.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr"
    
'   The contents of the local variable (objwks) is the address of the global object but byte-reversed (little endian)
'   Load the reference into a LongPtr type and read back the byte-corrected (big endian) value of the reference
    Mem_Copy objwksPtr, ByVal VarPtr(objwks), PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)

'   Can also do this passing objwks as ByRef (default)
    Mem_Copy objwksPtr, objwks, PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)

'   Or like this...
    Mem_Copy ByVal VarPtr(objwksPtr), objwks, PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)
    
End Sub

Output:

Code:

Sheet1  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x002AEF90 : Contents: 0x78052418    using varPtr
objwks  points to: 0x18240578
objwks  points to: 0x18240578
objwks  points to: 0x18240578

There are also some examples of how to call Mem_Copy (RtlMoveMemory) that might help to understand why it crashes.

And here you can see how the local variable's value is zero when its set to nothing...

Code:

Sub getReference2()
#If Win64 Then
    Const PTR_LENGTH As Long = 8
#Else
    Const PTR_LENGTH As Long = 4
#End If
Const coBytes As Long = 4
Dim objwks As Worksheet, lptr As LongPtr, objName As String, objwksPtr As LongPtr

    Debug.Print
'   Sheet1 Object reference and first coBytes bytes
    lptr = objPtr(Sheets(1)): objName = "Sheet1 "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
    
'   Using varPtr, can see the local variable is set to nothing.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr, set to Nothing"
    
'   Using objPtr, can't diferentiate because the reference is resolved before reporting
    Set objwks = Sheets(1)
    lptr = objPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using objPtr"
        
'   Using varPtr, can diferentiate.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr"
    
'   The contents of the local variable (objwks) is the address of the global object but byte-reversed (little endian)
'   Load the reference into a LongPtr type and read back the byte-corrected (big endian) value of the reference
    Mem_Copy objwksPtr, ByVal VarPtr(objwks), PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)
    
End Sub

Output:

Code:

Sheet1  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x002AEF90 : Contents: 0x00000000    using varPtr, set to Nothing
objwks  : Address: 0x18240578 : Contents: 0xC0EA4C18    using objPtr
objwks  : Address: 0x002AEF90 : Contents: 0x78052418    using varPtr
objwks  points to: 0x18240578

Quote:

Originally Posted by whatsup

2. If memory is freed, we might find "cleared to zero", but there might be something else assigned to the pointer. Up to now it only occured when testing with pointers to a Long-variable, but I think it always can happen. Therefore we better compare - keeping in mind point 1

If the memory is set to zero or overwritten, it makes no difference; it is still clear evidence that the variable's memory space has been released. And that's been my point all along...

In summary, I cant see any reason to set local objects to nothing at the end of a sub. Even if there is an error and the sub doesn't complete normally, or if there are structures that fool the reference counting, everything is still cleaned up in the former and the set to nothing does nothing extra in the latter.

I learned a lot from this exercise and added some nice routines to my library, so this has been most interesting!

For anyone who wants to run the above routines, here is a .bas file with everything you need... well its .txt but just need to change it to .bas and you can import into a module.

whatsup · #40 06-22-2014, 06:05 PM

Sorry for the delay, but i kept thinking and thinking ...

Point1: Post #35
I basically share your opinion about the application crash on the attempt to Mem_copy something what doesn't match a 4 Bytes Code.
By accident I choose first a range-object (and experienced the crashes), afterwards I changed to a worksheet-object (avoiding crashes I didn't any mem-copy to an object or a variant). I was just looking at the numbers of ObjPtr which are always the same whether you obtain them directly from the object
ObjPtr(Sheet1)
or from the variable
ObjPtr(objwks)
Therefore I wondered, why did excel crash, because even with the variable gone I ask at the end of the Copy of ObjPtr(Range("A1")), and since the Range still exists and hasn't changed, what's the problem to copy it?
It took me a long time to figure out there is a difference between Range and Worksheet:
Whereas Worksheet keeps his pointer throughout the application, this isn't the case with Range. A Range changes its pointer, furthermore the variable (objRange) set to the Range is assigned another pointer.

I said the choice was by accident but actually I'm glad about the choice because otherwise it would have led in a complete different direction...

Point2:
I regret the lack of knowlegde about the architecture of objects and what they look like in memory. I really do, probably it would make things a lot easier knowing about this things.

Point3:

Quote:

If the memory is set to zero or overwritten, it makes no difference; it is still clear evidence that the variable's memory space has been released. And that's been my point all along...

I would feel more comfortable if evidence would present itself more obvious, for example in just showing only zeros (as it does with a simple class). Why can't excel give a pointer some rest after he did a job

But Yes, I agree, at least for the moment let's depend on the change.

Point4:
According to Point 4, I agree, memory is freed whether or not the variable is set to Nothing (at least valid for the macros I tried up to now). A surprise to me: Classes are destroyed without explicity destroying them.
But I don't agree on this

Quote:

...or if there are structures that fool the reference counting...

If you refer to the example with circular references within classes, that's one thing which the example establishes most clearly: None of them get destroyed.

Attached the file including 4 tests - as well the CircRef - with some kind of summary.

CoolBlue · #41 06-22-2014, 07:37 PM

Quote:

Originally Posted by whatsup

Sorry for the delay, but i kept thinking and thinking ...

Point1: Post #35
I basically share your opinion about the application crash on the attempt to Mem_copy something what doesn't match a 4 Bytes Code.
By accident I choose first a range-object (and experienced the crashes), afterwards I changed to a worksheet-object (avoiding crashes I didn't any mem-copy to an object or a variant). I was just looking at the numbers of ObjPtr which are always the same whether you obtain them directly from the object
ObjPtr(Sheet1)
or from the variable
ObjPtr(objwks)
Therefore I wondered, why did excel crash, because even with the variable gone I ask at the end of the Copy of ObjPtr(Range("A1")), and since the Range still exists and hasn't changed, what's the problem to copy it?
It took me a long time to figure out there is a difference between Range and Worksheet:
Whereas Worksheet keeps his pointer throughout the application, this isn't the case with Range. A Range changes its pointer, furthermore the variable (objRange) set to the Range is assigned another pointer.

I said the choice was by accident but actually I'm glad about the choice because otherwise it would have led in a complete different direction...

I'll have to take some time to think about that, but they are interesting observations...
I'm surprised that the crashMem_Copy routine in my previous post crashes excel because the destination pointer is fine. Why crash when the source pointer is set to zero? Its only a read... Any thoughts?

Quote:

Originally Posted by whatsup

I regret the lack of knowlegde about the architecture of objects and what they look like in memory. I really do, probably it would make things a lot easier knowing about this things.

Well, clearly you are not alone! Nobody seems to have any knowledge about this, at least I cant find any.
But your link was very helpful, I've seen the byte comb site before but it was good to be reminded of it, it informed my thinking on this.

Quote:

Originally Posted by whatsup

I would feel more comfortable if evidence would present itself more obvious, for example in just showing only zeros (as it does with a simple class). Why can't excel give a pointer some rest after he did a job

But Yes, I agree, at least for the moment let's depend on the change.

It seems to me that the Erase Method in VBS's "garbage collection" system does indeed set the first 8 bytes to zeros. The point is, after a crash and re-start of another routine, a LOT has changed and the space will surely have been used for something else. So the zeros have been over-written.

Quote:

Originally Posted by whatsup

According to Point 4, I agree, memory is freed whether or not the variable is set to Nothing (at least valid for the macros I tried up to now). A surprise to me: Classes are destroyed without explicity destroying them.

Yes, I saw from one of your early quotes that you would be surprised about the classes being destroyed and I wanted to give you clear evidence about that and that's why I did the class example.
Its quite simple really, all object life cycle is managed by reference counting: when the reference count goes to zero, the object is scheduled for deletion. There are no no exceptions.
I guess this will be a background process that is run when VBARuntime has free time and it will just scan the reference count field in the object table and "Erase" those with zero reference. The ERASE method will not delete the object from the object table, but it will release any memory used by the object's structure.

Quote:

Originally Posted by whatsup

But I don't agree on this
If you refer to the example with circular references within classes, that's one thing which the example establishes most clearly: None of them get destroyed.

Attached the file including 4 tests - as well the CircRef - with some kind of summary.

OK, now we get to the main point

The full quote on this is:

Quote:

Originally Posted by CoolBlue

In summary, I cant see any reason to set local objects to nothing at the end of a sub. Even if there is an error and the sub doesn't complete normally, or if there are structures that fool the reference counting, everything is still cleaned up in the former and the set to nothing does nothing extra in the latter.

Sorry, the language is a bit unclear so I've added emphasis...

I am saying that setting to nothing does nothing extra in these cases. Im saying it makes no difference. The problem will still be there.
I am arguing against using set to nothing remember?

As you can see in your example, the result after run-time is the same if you have

Code:

Set objcls1 = Nothing

Or

Code:

'Set objcls1 = Nothing

THAT is my point: you cannot fix these issues with set to nothing. One of the links offered by my friend @macropd as a reason to use set to nothing was referring to these structures and my point was and still is that set to nothing makes no difference.

So, the two points I was trying to argue against are:

Your second point in post#7

Quote:

Originally Posted by whatsup

- The main reason: There are different types of objects, for instance classes you must unload manually; if creating a new instance of an object (whenever using the keyword "New") you have to unload it manually as well...

The claim by @macropod in post#7 that this link was supporting the use of set to nothing at the end of a sub as a way to manage (for example) objects with circular references.

I think we do agree on those...

And thanks for the attachment... your documentation style is much nicer than mine, I will study it and improve my ways!

whatsup · #42 06-22-2014, 10:15 PM

Quote:

In summary, I cant see any reason to set local objects to nothing at the end of a sub. Even if there is an error and the sub doesn't complete normally, or if there are structures that fool the reference counting, everything is still cleaned up in the former and the set to nothing does nothing extra in the latter.

Ok, that way I agree. Still, if short of memory you can deallocate memory earlier using Set to Nothing. But otherwise it's done automatically when leaving the sub. That's shown nicely in Run11, when you skip the part Set to nothing, the Terminate-Event of the class is still triggered (but I see you mentioned this already in your post)

And I remember very well that you argue against - therefore we are here

But I enjoy. That way I learn quite a lot about memory, which really is due in regard that obviously MS has improved on the subject.

Yeah, the VarPtr of an object - honestly I don't know if it makes sense. I tried to figure out whether the Mem_ReadHex of VarPtr points to the ObjPtr by converting the HexToDec, but was disapointed by the result. Even worse HexToDec comes up with negative figures, so what shall we make of this? Maybe Val("&H"&...) isn't the right tool?

Quote:

Why crash when the source pointer is set to zero? Its only a read... Any thoughts?

I don't know, maybe because it doesn't exist a pointer "Zero" within memory? Maybe "Zero" makes a program turn off the track ways before, but we are interfering far behind the curtain where nothing is provided anymore to reject the attempt? In fact you cannot copy anything if the pointer is Zero, therefore I just excluded it from any attempt.

Documentation is a selfish thing - I know after several days I will end up with x files not knowing anymore what I intended to achieve, just testing, playing, ...

But one thing is sure, the time I used formatting the immediate window, I rather would have spent for another output. But for the moment it's ok.

Me as well, I spent a lot of time searching the web for information about "memory" and related words, but google misunderstands what I'm looking for, and mostly comes up with things I'm not interested in.

CoolBlue · #43 06-23-2014, 02:38 AM

Quote:

Originally Posted by whatsup

Ok, that way I agree. Still, if short of memory you can deallocate memory earlier using Set to Nothing. But otherwise it's done automatically when leaving the sub. That's shown nicely in Run11, when you skip the part Set to nothing, the Terminate-Event of the class is still triggered (but I see you mentioned this already in your post)

OK, cool

And yes, I agree, its a good idea to set objects to nothing earlier if you don't need them anymore... but if you are already at the end of the sub then... yep, we agree on that.

Quote:

Originally Posted by whatsup

And I remember very well that you argue against - therefore we are here

But I enjoy. That way I learn quite a lot about memory, which really is due in regard that obviously MS has improved on the subject.

I am enjoying the learning also!

Quote:

Originally Posted by whatsup

Yeah, the VarPtr of an object - honestly I don't know if it makes sense. I tried to figure out whether the Mem_ReadHex of VarPtr points to the ObjPtr by converting the HexToDec, but was disapointed by the result. Even worse HexToDec comes up with negative figures, so what shall we make of this? Maybe Val("&H"&...) isn't the right tool?

Ah, this one I can explain...
The memory image is in a format called little endian. Intel-based systems store bytes little end first so the least significant byte is on the left. You have to flip the bytes to see the true hex value...
If you look at the last section of getReference2() in my previous post you will see how that I move the contents of the local pointer (the address of Sheet1) into another LongPtr type called objwksPtr. I can then print out the hex value of that pointer and VBA will interpret it correctly.

Code:

01 Sheet1  : Address: 0x1D6DA618 : Contents: 0xC0EA4C18    using objPtr
02 objwks  : Address: 0x002AEF90 : Contents: 0x00000000    using varPtr, set to Nothing
03 objwks  : Address: 0x1D6DA618 : Contents: 0xC0EA4C18    using objPtr
04 objwks  : Address: 0x002AEF90 : Contents: 0x18A66D1D    using varPtr
05 objwks  points to: 0x1D6DA618

Line 04 is from this...
'   Using varPtr, can diferentiate.
'   The local Worksheet Object is treated as a reference, not an Object: the reference is not resolved
    lptr = VarPtr(objwks): objName = "objwks "
    Debug.Print PointerToSomething(objName, lptr, coBytes) & vbTab & "using varPtr"
It produces this image of the contents of the objwks reference
    0x18A66D1D

Line 05 is from this
'   The contents of the local variable (objwks) is the address of the global object but byte-reversed (little endian)
'   Load the reference into a LongPtr type and read back the byte-corrected (big endian) value of the reference
    Mem_Copy objwksPtr, ByVal VarPtr(objwks), PTR_LENGTH
    Debug.Print objName & vbTab & "points to: " & "0x" & HexPtr(objwksPtr)
It produces this image of the contents of the objwks reference
    0x1D6DA618
And this is the correct address of the Sheet1 Object

But in fact they are the same number but in different formats...
    1D6DA618 18A66D1D

I just did it like that for illustration, the other way is to shuffle the bytes in the Mem_ReadHex routine. I added a sub called Mem_ReadHex_Endian_Aware to your Mod00_PublicFunctionsAPI module and incorperated it into your pointerToSomething routine in the attached version of your spreadsheet.
Ive also attached a couple of pics to try to make it clear...

CoolBlue · #44 06-23-2014, 06:35 AM

I am attaching a better version for handling the endian-ness...
This version is actually endian-agnostic: I let VBA figure it out and we always get the correct answer.
The routine is called Mem_ReadHex_Words. It also allows for 4, 8 or 16 bit words which are required by some data structures.
I use the idea of loading the memory image, one word at a time, into a LongPtr Type and then convert the LongPtr to hex. All the underlying byte swapping is taken care of by VBA automatically.

whatsup · #45 06-23-2014, 09:00 AM

Quote:

Ah, this one I can explain...

Ey, that's excellent news. Now that you mention it I remember reading about it on the byte comb page, but didn't bother pay attention to it

I will have a look at your files later on, is to say in the evening, still got some other work to do ...

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Wierd "script code" in a downloaded .doc file	CNBarnes	Word	2	10-18-2012 02:07 AM
replace data from variable with "sub and super script" from excel to word by vba	krishnaoptif	Word VBA	9	06-22-2012 05:08 AM
How to choose a "List" for certain "Heading" from "Modify" tool?	Jamal NUMAN	Word	2	07-03-2011 03:11 AM
Rules and Alerts: "run a script"?	discountvc	Outlook	0	06-15-2010 07:36 AM
An "error has occurred in the script on this page"	decann	Outlook	8	09-03-2009 08:54 AM