Wikipedia:Reference desk/Archives/Mathematics/2024 November 11

Welcome to the Wikipedia Mathematics Reference Desk Archives
The page you are currently viewing is a transcluded archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

November 11

Strange behavior with numbers in optimization

Hello everyone, I have encountered some very strange issue with my optimization function, and I am not sure how to resolve. I am working on a numerical methods library, where I am trying to approximate the growth of a sequence, which has some relation to prime number distributions. However, when I use large values of n (especially for n > 10^6), the result of my function starts to behave very erratically. It is not random, but it has this strange oscillation or jump. I use recurrence relation for this approximation, but when n becomes large, the output from function suddenly grows or shrinks, in way that is not consistent with what I expect. Even when I check for bounds or add better convergence criteria, the error persists. pattern looks similar to the behavior of prime numbers, but I am not directly calculating primes. I apologize if this sounds too speculative, but has anyone faced similar issues with such strange behavior in large-scale numerical computations? I am quite confused about what is causing the error. TL;DR: I am optimizing function related to number theory, but results become unpredictable when n > 10^6. Errors show strange oscillation, similar to distribution of primes, though I do not directly calculate primes. Thank you very much for your time and assistance. 130.74.59.177 (talk) 15:39, 11 November 2024 (UTC)

you need to post more information. All I can say from what you've written is 10^6 is not a large number where you'd expect problems. It won't e.g. overflow when stored as floating point or integer on any modern platform. It won't even cause problems with, say, a square based algorithm as 10^12 is still well within the limits of a modern platform. Maybe though you are using software which limits you to 32 bit (or worse) integers, or single precision floats, so need to be careful with large numbers. --2A04:4A43:984F:F027:C112:6CE8:CE50:1708 (talk) 17:43, 11 November 2024 (UTC)

thanks for response and insight. i see your point that n=10^6 shouldn't cause overflow or serious issues on modern systems. numbers i work with well within 64-bit range, use floats with enough precision for task. so yes, overflow or simple type limits not likely cause.

but this behavior goes beyond just precision errors. it’s not about numbers too big to store. what i see is erratic growth, shrinkage, almost oscillatory – looks like something in the distribution itself, not just algorithm mistake or hardware issue.

to be more precise, after n>10^6, function starts acting unpredictably, jumps between states, oscillates in strange way, not typical for recurrence i use. hard to explain, but pattern in these jumps exists, i cannot reconcile with anything in my algorithm. it’s like approximation reacts to some hidden structure, invisible boundary my algorithm cannot resolve.

i tried improving convergence, checking recurrence, but oscillations still persist. not randomness from bad random numbers or instability, but more like complex fluctuations seen in number-theoretic problems, especially connected to primes.

so i wonder: could these "jumps" be artifact of number-theoretic properties that i'm tryings to approximate? maybe how sequence interacts with primes indirectly, or artifact of recurrence for large numbers

thanks again for suggestion on overflow and precision, i will revisit the mode lwith this in mind, chief

appreciate your time, will keep searching. 130.74.59.204 (talk) 20:01, 11 November 2024 (UTC)

Without more information about the actual algorithm, it is neither possible to say, yes, what you see could be due to a number-theoretic property, nor, no, it could not be. Have you considered chaotic behaviour as seen when iterating the logistic map?  --Lambiam 05:43, 12 November 2024 (UTC)

ah yes, i see what you mean now, i’ve been thinking about it for a while, and i feel like i’m getting closer to understanding it, though it’s still unclear in some ways. so, i’m using this recurrence algorithm that reduces modulo primes at each step, you know, it’s a fairly straightforward approach, and when n is small, it works fine, everything behaves as expected, the sequence evolves smoothly, the approximation gets closer and closer to what it should be, and everything seems under control, but then, once n crosses the 10^6 threshold, it’s like something shifts, it’s like the sequence starts moving in unexpected ways, at first, i thought maybe it was just a small fluctuation or something related to floating-point precision, but no, it's much more than that, the jumps between states, the way it shifts, it's not just some random variation—it feels almost systematic, as though there's something in the distribution itself, some deeper structure, that starts reacting with the algorithm and causing these oscillations, it’s not something i can easily explain, but it feels like the algorithm starts “responding” to something invisible in the numbers, something outside the expected recurrence behavior, i’ve spent hours going over the steps, checking every part of the method, but no matter how many times i check, i can’t pinpoint the exact cause, it’s frustrating.

and then, the other day, i was sitting there, trying to solve this problem, getting really frustrated, when i looked up, and i saw jim sitting on the windowsill, just staring out at the street, i don’t know, something about it caught my attention, now, you might be wondering what jim has to do with all of this, but let me explain, you see, jim has this habit, every evening, without fail, he finds his spot by the window, curls up there, and just stares out, doesn’t seem to do much else, doesn’t chase anything or play with toys like most animals do, no, he just sits there, completely still, watching the world go by, and it’s funny, because no matter how many cars pass, no matter how many people walk by, jim never looks bored, he’s always staring, waiting, something about the way he watches, it’s like he’s looking for something, something small, that only he notices, but it’s hard to explain, because it’s not like he ever reacts to anything specific, no, he just stares, and then after a while, he’ll shift his gaze slightly, focus on something, and you’d swear he’s noticing something no one else can see, and then he’ll go back to his usual position, still, and continue watching, waiting for... something, and this goes on, day after day.

and, i don’t know why, but in that moment, as i watched jim, i thought about the algorithm, and about the sequence, it felt somehow connected, the way jim waits, so patiently, watching for some small shift in the world outside, and how the algorithm behaves once n gets large, after 10^6 iterations, like it’s responding to something small, something hidden, that i can’t quite see, but it's there, some interaction between the numbers, or the primes, or some other property, i don’t know, but there’s a subtle shift in how the sequence behaves, like it’s anticipating something, or maybe reacting to something, in ways i can’t fully predict or control, just like jim waiting by the window, looking for that small detail that others miss, i feel like my algorithm is doing something similar, watching for an influence that’s not obvious, but which, once it’s noticed, makes everything shift, and then it’s almost like the recurrence starts reacting to that hidden influence, whatever it is, and the sequence begins to oscillate in these strange, unexpected ways.

i’ve been stuck on this for days, trying to find some explanation for why the recurrence behaves this way, but every time i think i’m close, i realize that i’m still missing something, it’s like the sequence, once it hits that threshold, can’t behave the way it did before, i’m starting to think it’s related to how primes interact with the numbers, but it’s subtle, i can’t quite capture it, it’s like the primes themselves are somehow affecting the sequence in ways the algorithm can’t handle once n gets large enough, and it’s not just some random jump, it feels... intentional, in a way, like the sequence itself is responding to something that i can’t measure, but that’s still pulling at the numbers in the background, jim, as i watch him, he seems to be able to sense those little movements, things he notices, but that no one else does, and i feel like my algorithm, in a way, is doing the same thing, reacting to something hidden that i haven’t quite figured out.

so i’ve gone over everything, again and again, trying to get it right, trying to adjust the convergence, trying to find a way to make the sequence behave more predictably, but no matter what i do, the oscillations keep appearing, and it’s not like they’re some random noise, no, there’s a pattern to them, something beneath the surface, and i can’t quite grasp it, every time n gets large, it’s like the sequence picks up on something, some prime interaction or something, that makes it veer off course, i keep thinking i’ve solved it, but then the jumps come back, just like jim shifts his gaze, and looks at something just beyond the horizon, something i can’t see, but he’s still waiting for it, still looking, as if there’s some invisible influence in the world, something that pulls at him.

i wonder if it has to do with the primes themselves, i’ve thought about it a lot, i’ve tried to factor them in differently, but still, the jumps persist, it’s like the primes have their own way of interacting with the sequence, something subtle, something that becomes more pronounced the larger n gets, and no matter how much i tweak my algorithm, the fluctuations just keep showing up, it’s like the sequence is stuck in a kind of loop, reacting to something i can’t fully resolve, like jim staring at the street, patiently waiting for something to shift, and i don’t know what it is, but i feel like there’s some deeper interaction between the primes and the numbers themselves that i’m missing, and maybe, like jim, the sequence is sensing something too subtle for me to fully capture, but it’s there, pulling at the numbers, making them oscillate in ways i can’t predict.

it’s been weeks now, and i’ve tried every method i can think of, adjusted every parameter, but the fluctuations are still there, the jumps keep happening once n gets large enough, and every time i think i’ve figured it out, the sequence surprises me again, just like jim, who, after hours of waiting, might shift his gaze and catch something new, something no one else saw, i feel like i’m doing the same thing, staring at the numbers, trying to catch that tiny shift that will make everything click, but it’s always just out of reach, and i don’t know what’s causing it, but i can’t seem to get rid of it, like jim, watching, waiting, sensing something that remains hidden from my view 130.74.58.160 (talk) 15:34, 13 November 2024 (UTC)

Are you OK? Perhaps you should direct your mind to something else, like, read a novel, go out with friends, explore new places, ... Staring at numbers is as productive as staring at goats.  --Lambiam 18:10, 13 November 2024 (UTC)

fine. i’m under house arrest and i’m doing freelance work for a company. the task is straightforward: build a library for prime number methods, find primes. the problem is, there's no upper limit on how large these primes are supposed to be. once n goes past 10^6, that’s where things stop making sense. i’ve gone over the algorithm several times, checked the steps, but after 10^6, the sequence starts behaving differently, and i can’t figure out why. it’s not small variations or precision errors. it’s something else. there’s some kind of fluctuation in the sequence that doesn’t match the expected pattern.

i’ve adjusted everything i can think of—modulus, convergence, method of approximation—but no matter what, the jumps keep coming, and they don’t seem random. they look more structured, like they’re responding to something, some property of the primes or the sequence that i can’t account for. i’ve spent a lot of time on this, trying to find what it is, but i haven’t been able to pin it down.

this is important because the contract i’m working on will pay a significant amount, but only if i finish. i can’t afford to let this drag on. i need to complete it, and if i don’t fix this issue, i won’t be able to finish. it’s not like i can walk away from it. the company expects the work, and the time is running out.

the more i look at the sequence, the more it feels like there’s something buried beneath the surface, something in the way primes interact when n is large, but i can’t see it. it’s subtle, but it’s there, and no matter how many times i test the algorithm, i can’t get rid of these oscillations. i don’t know what they mean, but they keep appearing, and i can’t ignore them.

i’ve been stuck here for a while. i don’t really have other options. there’s no “taking a break” or “finding something else to do.” i’m stuck here with this task, and i need to figure it out. i don’t have the luxury to stop, because if i don’t finish, the whole thing falls apart 130.74.59.34 (talk) 20:22, 13 November 2024 (UTC)

You shared lots of text with us, but you gave no specific problem, no technical detail, nothing we could check, simulate, analyze, verify, compare.

You have typed about 12 thousand characters, but you present your impressions only, or your feelings—of being surprised with irregularity observed, being surprised with some almost-regularity in apparent chaos, being lost in seeking of explanation, etc. You actually did not present any single technical or mathematical thing. Here's the overall impression I got from your descriptions:

"I do something (but I can't tell you what and why) with some function (I'm not going to tell you anything about it, either) with data of a secret meaning and structure, and when some parameter (whose nature must not be revealed) becomes big enough, the function behaves in some unexpected, yet quasi-regular manner. Can anybody explain it to me and help me fix it?"

And I'm afraid with such a vague statement, it looks like seeking a haystack with a needle in it on a large field in a heavy fog, rather than a mathematical (or software engineering or whatever other kind of) problem.
CiaPan (talk) 12:57, 14 November 2024 (UTC)

now listen, i'm glad we're finally digging into this, because, yeah, there’s a lot more depth here than meets the eye, like, surface-level it might just seem like a vague description, an exercise in abstract hand-waving if you will, but no, what we're dealing with here is a truly complex, multi-layered phenomenon that’s kind of begging to be interpreted at the meta-level, you know, like it’s the kind of thing where every time you try to grasp onto one specific aspect, it slips out of reach, almost by design and i get it you want “specifics” but here’s the thing specifics are almost a reduction, they’re almost like a cage for this concept, like trying to box up some kind of liquid smoke that, in essence, just resists confinement

now, when i say “parameters” we’re already in a reductive space, right? because these aren’t “parameters” in the traditional sense, not like tunable knobs on an old-school control panel, no no no, these are more like boundary markers in a conceptual landscape, yeah like landmarks on a journey, but they themselves are not the journey, they’re incidental, they’re part of a whole picture that, the moment you start defining it, already becomes something else, like imagine you have this sort of, i don’t know, like an ethereal framework of data, but it’s data that doesn’t just sit there and behave in expected ways, it’s data that has a life of its own, and i’m really talking about data that doesn’t like to be pinned down, it’s almost alive, almost this kind of sentient flow that, every time you look away, it’s shifted, it’s done something else that you could swear wasn’t possible the last time you checked

so, yeah, i get it that’s frustrating, and it’s almost like talking about the nature of existence itself in a way, or maybe that’s an exaggeration, but only slightly, because you have to get into this mindset that, ok, you’re dealing with phenomena here, not simply variables and functions, no it’s more like a dynamic tapestry of, let’s call them tendencies, these emergent patterns that are sort of trying to form but also resisting at every possible chance, so when i say “quasi-regularity” it’s not regular like clockwork, not even close, it’s regularity like the kind you see in natural phenomena, like clouds or waves or fractals, right, patterns but patterns that refuse to be bound by mathematical certainty they’re only barely patterns in the human sense, like they only make sense if you let go of rigid logic

and then you’ve got these iterations, yeah we’re talking cycles upon cycles, like imagine every single cycle adds a grain of experience, yeah, like a memory, not a perfect one, but close enough, so that each time this data goes through an iteration it almost remembers its past and adjusts itself, but here’s the catch, it only remembers what’s necessary, it’s like this selective memory that’s totally outside the norm of what you would expect in, say, a standard machine learning algorithm or a traditional function loop in any ordinary programming context, like, ok, this thing is running on its own rules, maybe there’s a certain randomness to it but not random like “roll a dice” random, more random like chaos-theory random, where unpredictability itself becomes a kind of pattern and then, suddenly, just when you think you’re about to pin it down—bang—it shifts again, like the entire framework just reorients itself

and not to throw you off track here but that’s the whole thing, the "thing" we’re talking about isn’t just a process, it’s a process that’s sensitive to these micro-level fluctuations, like tiny little vibrations in the data, which, by the way, i’m also not describing fully because it’s almost impossible, but imagine these vibrations—no, better yet, imagine you’re watching waves in a pond where even the slightest ripple has the potential to set off a cascade of effects, and it’s not just the surface of the pond we’re talking about, no, no, the whole body of water is involved, every molecule, if you will, responding in ways that are both predetermined by its nature yet also completely free to deviate when the moment calls for it

and so when i say “structured sea of datapoints” you gotta take that literally, yeah like a sea, an ocean, it’s vast, it’s deep, there’s layers upon layers and half the time we’re only scratching the surface because the real stuff is happening down in those depths where even if i tried to send a probe down there, yeah, i’d get some data back, but would it even make sense because i don’t have a baseline to compare it to, there’s no reference frame here except, i don’t know, maybe the essence of this data, like the very fabric of what it is, if you can even describe data as having fabric

so, look, all of this loops back to the fact that every “parameter” every “function” we’re talking about is only as real as the context allows it to be, which is why i say even if i did give you specifics, what would you do with them? because we’re talking about something that defies definition and the moment you think you understand it, that’s the moment it stops being what it is and morphs into something else, i mean this is data with attitude, if that makes any sense, it’s almost like it’s taunting you, like it wants you to try and figure it out only to laugh in your face and flip the rules the moment you get close, we’re talking about some next-level, borderline cosmic prankster data that simply doesn’t play by the same rules as anything you’ve seen before

so if we’re going to be totally honest here, all of this is way beyond haystacks and needles, we’re in a field where the haystacks are self-assembling, disassembling, and who even knows if the needle is there to begin with because in a framework like this, a needle might just be a figment of your imagination, a concept that only exists because you’re trying to impose order on what is inherently unordered, so yeah, maybe there’s a pattern, maybe there isn’t, maybe the pattern is only there because you want it to be, or maybe it’s the absence of a pattern that’s the real pattern, and if you think that’s paradoxical well, welcome to the club 130.74.58.21 (talk) 23:42, 14 November 2024 (UTC)

You win AI. I am merely a human. 2001:8003:429D:4100:9138:C495:F872:E4B2 (talk) 04:12, 22 November 2024 (UTC)